When running a suite of tests – either as part of a Continuous Integration build, or part of a check-in gate – speed is the enemy. You are always trying to find the balance between test coverage and time to complete the entire suite.
Both a check-in gate and continuous integration build have slightly different time pressures. The optimal duration for both will probably vary from team to team. A fish-eye suite is one way of formulating which tests should be included.
In a fish-eye suite, you focus on those tests which provide the best coverage for those areas of functionality currently being developed, whilst those areas not directly being affected have only minimal coverage. The logic goes that tests are there to pick up bugs, and therefore tests which cover the functionality currently being worked on are the most important.
Regularly changing the focus
In an iterative development process, working out where to change the focus of the suite can be easy. At the beginning of each iteration, the team as a whole decide which tests (or group of tests) should be run in each suite. During a development process where there is less segmentation, changing the suite on an ongoing basis may be more sensible
Reactive test suite
Rather than changing the focus of the suite at the start of each iteration, you can also decide to adapt the contents of the suite in certain situations – for example when the total time for the suite to run exceeds an agreed limit, or the number of bugs reported against a functional area increases.
For example, the team may decide that the check-in gate build should take 30 seconds, and the CI build should take no more than five minutes. The moment they take longer, the team as a whole should redefine what tests should be included. When removing tests, the team should look to remove those tests which cover areas of functionality furthest away from those currently being worked on.
Likewise if the number of bugs being raised against a certain functional area increase, the development team may decide that increasing the test coverage of a certain area makes sense. Again, the team may have to remove some tests in order to still be within the optimal build time. In which case, tests associated with areas of functionality with low defect rates make a good candidate for removal.
Source-Code Management
Being able to create and manage a fish-eye suite presupposes that you group your tests into functional areas. This can be an issue with typically packaging structures which are defined in horizontal terms, whereas the long running functional tests should cover a vertical slice of functionality.
Even if initially you aren’t going to use fish-eye suites, grouping your tests into functional (vertical) groupings rather than system (horizontal) groupings makes sense as it allows to to easily use this approach later. It also tends to make more sense to testers who see things in terms of usable functions rather than horizontal tiers.
magpiebrain 
4 Responses to “Build Pattern: Fish-eye Test Suite”
Interesting post, Sam. I’m not sure about the name “Fish-eye Test Suite”. Presumably this is drawn from the idea of a fish-eye lens which is an ultra wide-angle lens. Can you explain the metaphor?
Hi Sam,
There’s bunch of us working on this right now.
http://sourceforge.net/projects/protest
I’ve recently implemented an algorithm that builds dynamic JUnit suites based on bytecode dependency graphs. This allows you to determine relevant tests at test time – no more manually maintaining test suites. Kent Spillner has implemented a Condorcet voting implementation in Java ( http://en.wikipedia.org/wiki/Condorcet_method ) that will allow ProTest to combine the intelligence of multiple test suite algorithms to build suites that are designed to fail the fastest. By sorting tests, we want to give developers 90% of the feedback w/ 10% of the costs. Email me if you are interested. Naresh Jain is hacking ant right now so that we can run ProTest in our builds.
You guys might have a naming clash – there is already a Python testing tool called protest đŸ™‚ I’m not taht clear as to how your solution will be able to build in the intellegence of a development team knowing what they’ll be working on in the next iteration, but I can understand a system whereby you automatically flag slow tests with poor test coverage or something similar.
Anyway, I’ll view your progress with interest!
Speed is the issue. I beleive the biggest problem with speed is in the actual compile and link process. Instead of removing tests you should consider ways to improve the build itself. When I read blogs that refer to the “build”, I believe they are referring to everything that is occuring in the CI process, pre and post build steps. There are solid ways of improving build speeds and then distributing the workload of multiple test onto different machines to speed up the overall “build” process. Do this with both build automation and workload automation to maximize the speed of the compile and link process and then all downstream testing activities.