Sam Newman's site, a Consultant at ThoughtWorks

Matt Riable has encountered a performance issue with his JUnit tests, and is “advocating the use of static data(Make your JUnit Tests run faster when using Spring)”: to construct his Spring @ApplicationContext@. I think in this specific instance he might be justified in his choice (I’m nothing if not pragmatic) but I am always extremely cautious in allowing the use of a shared environment between tests.

Lets look at the crux of Matt’s problem – by the book you should define the environment for a test within @setUp()@, and should clear it up in @tearDown@. @setUp@ is called before each @test@ call, and @tearDown@ after, in order to isolate each test from the other. In Matt’s case, the @setUp@ call is quite slow – the creation of his @ApplicationContext@ involves file IO and XML parsing, not the fastest of processes. So why is @setUp@ called prior to each test? Let’s look at a very simple example:

public ExampleTest extends TestCase {
  private SharedObject sharedObject;

  public void setUp() {
    sharedObject = new SharedObject();

  public void testOne() {
    //run some test using sharedObject

  public void testTwo() {
    //run some test using sharedObject

Imagine if @setUp()@ is only called one for the whole @testCase@ – what if @testOne@ or @testTwo@ change the state of @sharedObject@? If @testOne@ is run first, then @testTwo@ will be reliant on the state of @sharedObject@ manipulated by @testOne@ – can you be sure @testTwo@ will still work if you run it first? Many of us run our JUnit tests directly in our IDE, and we have no control over the order in which tests are run. By sharing potentially mutable data between tests we are exposing ourselves to the possibility of variable results based on the order in which tests are run.

Now I’m not advocating the notion that shared data should never be used – for example I cannot see a problem sharing immutable data between tests. However think very carefully about sharing any data which has the capacity to be altered by one of the tests – its important to balance the benefits of instantiating Objects once for all of the tests against the potential risk of poluting the state of one test with the operations of another.


4 Responses to “The importance of Unit test isolation”

  1. Robert Watkins

    One very simple technique in this case (of expensive setup) is to use a TestSetup decorator for the TestSuite to load the data _once_. Then, the individual TestCase setup methods can simply do an in-memory clone of the parsed data.

    This gives test-case isolation, while preserving speed.

    There’s also a TestCase implementation I saw once which deliberately runs the test methods in a randomized order.

  2. Sam Newman

    Excellent information, thanks Robert. Helpful posts like this is why I put up with all the hassle that having comments can cause 🙂


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Basic HTML is allowed. Your email address will not be published.

Subscribe to this comment feed via RSS

%d bloggers like this: