| Home | Libraries | People | FAQ | More

# Boost Test Policies and Protocols

The Boost libraries are intended to be both reliable and portable. Every experienced programmer knows that means each library must be tested against a suitable number of test cases, on a wide range of platforms, and then tested again (regression tested) every time a change is made and before every release.

"Quality assurance based on a wide range of targeted tests" as one of the key answers to C.A.R Hoare's question "How did software get so reliable without proof."

## Regression test

Boost uses an automatic regression test suite which generates HTML compiler status tables.

## Test Policy

### Required

- Every Boost library should supply one or more suitable test programs to be exercised by the Boost regression test suite. In addition to the usual compile-link-run tests expecting successful completion, compile-only or compile-and-link-only tests may be performed, and success for the test may be defined as failure of the steps.
- Test program execution must report errors by returning a non-zero value. They may also write to stdout or stderr, but that output should be relatively brief. Regardless of other output, a non-zero return value is the only way the regression test framework will recognize an error has occurred. Note that test programs to be included in the status tables must compile, link, and run quickly since the tests are executed many, many, times.
- Libraries with time consuming tests should be divided into a fast-execution basic test program for the status tables, and a separate full-coverage test program for exhaustive test cases. The basic test should concentrate on compilation issues so that the status tables accurately reflect the library's likelihood of correct compilation on a platform.
- If for any reason the usual test policies do not apply to a particular library, an alternate test strategy must be implemented.

### Optional

- Use the Boost Test Library for the construction of simple tests.
- [Planned] Use the Boost Unit Test Library for the construction of more complex tests and test suites.
- The Boost Program Execution Library is also available for very simple tests, but the Boost Test Library is much preferred even for very simple tests.

## Suggested Protocol for Fixing Bugs or Adding Features.

- First, add regression test cases that detects the bug or tests the feature. Sometimes adding one case suggests similar untested cases, and they are added too.
- Second, for bugs, run the regression test and verify that the bug is now detected.
- Third, then, and only then, fix the bug or add the feature.
- Finally, rerun the full regression tests - sometimes the change breaks something else.

## History

The current regression test program was developed by Jens Maurer in December, 2000. The original regression test concept and program, including the automatic generation of HTML compiler status tables, was developed by Beman Dawes in June, 2000.

## Acknowledgements

Written by Beman Dawes. Jens Maurer, Paul Moore, Gary Powell and Jeremy Siek contributed helpful suggestions.

Revised 17 February, 2003