Test harness

In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.

Test harnesses allow for the automation of tests. They can call functions with supplied parameters and print out and compare the results to the desired value. The test harness is a hook to the developed code, which can be tested using an automation framework.

A test harness should allow specific tests to run (this helps in optimizing), orchestrate a runtime environment, and provide a capability to analyse results.

The typical objectives of a test harness are to:

• Automate the testing process.
• Execute test suites of test cases.
• Generate associated test reports.

These individual objectives may be fulfilled by unit test framework tools, stubs or drivers.^[1]

A test harness may provide some of the following benefits:

• Increased productivity due to automation of the testing process.
• Increased probability that regression testing will occur.
• Increased quality of software components and application.
• Repeatability of subsequent test runs.
• Offline testing (e.g. at times that the office is not staffed, like overnight).
• Access to conditions and/or use cases that are otherwise difficult to simulate (load, for example).

Alternative definition

An alternative definition of a test harness is software constructed to facilitate integration testing. Where test stubs are typically components of the application under development and are replaced by working components as the application is developed (top-down integration testing), test harnesses are external to the application being tested and simulate services or functionality not available in a test environment.

For example, when attempting to build an application that needs to interface with an application on a mainframe computer, but no mainframe is available during development, a test harness may be built to use as a substitute.

A test harness may be part of a project deliverable. It is kept separate from the application source code and may be reused on multiple projects. A test harness simulates application functionality; it has no knowledge of test suites, test cases or test reports. Those things are provided by a testing framework and associated automated testing tools.

The test harness will generally be specific to a development environment such as Java. However, interoperability test harnesses have been developed for use in more complex systems.^[2]

References

Further reading

• Pekka Abrahamsson, Michele Marchesi, Frank Maurer, Agile Processes in Software Engineering and Extreme Programming, Springer, 1 January 2009