When running a suite of tests – either as part of a Continuous Integration build, or part of a check-in gate – speed is the enemy. You are always trying to find the balance between test coverage and time to complete the entire suite.
Both a check-in gate and continuous integration build have slightly different time pressures. The optimal duration for both will probably vary from team to team. A fish-eye suite is one way of formulating which tests should be included.
In a fish-eye suite, you focus on those tests which provide the best coverage for those areas of functionality currently being developed, whilst those areas not directly being affected have only minimal coverage. The logic goes that tests are there to pick up bugs, and therefore tests which cover the functionality currently being worked on are the most important.
Regularly changing the focus
In an iterative development process, working out where to change the focus of the suite can be easy. At the beginning of each iteration, the team as a whole decide which tests (or group of tests) should be run in each suite. During a development process where there is less segmentation, changing the suite on an ongoing basis may be more sensible
Reactive test suite
Rather than changing the focus of the suite at the start of each iteration, you can also decide to adapt the contents of the suite in certain situations – for example when the total time for the suite to run exceeds an agreed limit, or the number of bugs reported against a functional area increases.
For example, the team may decide that the check-in gate build should take 30 seconds, and the CI build should take no more than five minutes. The moment they take longer, the team as a whole should redefine what tests should be included. When removing tests, the team should look to remove those tests which cover areas of functionality furthest away from those currently being worked on.
Likewise if the number of bugs being raised against a certain functional area increase, the development team may decide that increasing the test coverage of a certain area makes sense. Again, the team may have to remove some tests in order to still be within the optimal build time. In which case, tests associated with areas of functionality with low defect rates make a good candidate for removal.
Being able to create and manage a fish-eye suite presupposes that you group your tests into functional areas. This can be an issue with typically packaging structures which are defined in horizontal terms, whereas the long running functional tests should cover a vertical slice of functionality.
Even if initially you aren’t going to use fish-eye suites, grouping your tests into functional (vertical) groupings rather than system (horizontal) groupings makes sense as it allows to to easily use this approach later. It also tends to make more sense to testers who see things in terms of usable functions rather than horizontal tiers.