# Boost Test Library: Test Tools

Introduction
Example
Test/Sample Programs
Header Implementation Option
Rationale
Design
Also see: Execution Tools

## Introduction

The Boost Test Library's Test Tools supply several components to ease creation and maintenance of test programs.

- Header boost/test/test_tools.hpp supplies macros and function declarations to simplify detecting and reporting errors. While the functions can be called directly, the usual usage is via the convenience macros:

          BOOST_TEST( bool );            // continue after a failure
          BOOST_CRITICAL_TEST( bool );   // stop after a failure
          BOOST_ERROR( char* );          // continue after reporting error
          BOOST_CRITICAL_ERROR( char* ); // stop after reporting error

- A cpp_main() function is provided which calls a user-supplied test_main() function. The library supplied cpp_main() relieves users from messy error detection and reporting duties. The default implementation is layered on top of Execution Tools which supplies the program's main() function, and handles errors not specific to the Boost Test Tools.

Boost Test Tools are intended for fairly simple test programs. Execution Tools may be more suitable for production (non-test) programs. Unit Test Tools may be more suitable for complex test programs.

The boost/test_tools.hpp macros and functions are intended for test code rather than library or applications code, where throwing exceptions, assert(), or BOOST_STATIC_ASSERT() may be more suitable ways to detect and report errors.

## Example

The example program shows four different ways to detect and report an error in the add() function.

>     #include <boost/test/test_tools.hpp>    // see "Header Implementation Option"

>     int add( int i, int j ) { return i+j; }

>     int test_main( int, char *[] )  // note the name!
>     {
>       // six ways to detect and report the same error:
>       BOOST_TEST( add(2,2) == 4 );           // #1 continues on error
>       BOOST_CRITICAL_TEST( add(2,2) == 4 );  // #2 throws on error
>       if ( add(2,2) != 4 )
>         BOOST_ERROR( "Ouch...");             // #3 continues on error
>       if ( add(2,2) != 4 )
>         BOOST_CRITICAL_ERROR("Ouch...");     // #4 throws on error
>       if ( add(2,2) != 4 ) throw "Oops...";  // #5 throws on error
>       return add(2,2) == 4 ? 0 : 1;          // #6 returns error code
>     }

Approach #1 uses the BOOST_TEST macro, which displays an error message on std::cout that includes the expression that failed, the source file name, and the source file line number. It also increments an error count. At program termination, the error count will be displayed automatically by the Boost Test Tools supplied cpp_main() function..

Approach #2 using the BOOST_CRITICAL_TEST macro, is similar to #1, except that after displaying the error, an exception is thrown, to be caught by the library supplied cpp_main() function. This approach is suitable when writing a explicit test program, and the error would be so severe as to make further testing impractical. BOOST_CRITICAL_TEST differs from the C++ Standard Library's assert() macro in that it is always generated, and channels error detection into the uniform boost/test/test_main.cpp reporting procedure.

Approaches #3 and #4 are similar to #1 and #2 respectively, except that the error detection is coded separately. This is most useful when the specific condition being tested is not indicative of the reason for failure.

Approach #5 throws an exception, which will be caught and reported by the main() function supplied by either the Execution Tools or Test Tools. This approach is suitable for both production and test code, in libraries or not. The error message displayed when the exception is caught will be most meaningful if the exception is derived from std::exception, or is a char * or std::string.

Approach #6 uses a return value to inform the caller of the error. This approach is particularly suitable for integrating existing test code with the test tools library. Although it works fine with the Boost Program Execution or Test libraries, and is very useful for running existing code under them, most C++ experts prefer using exceptions for error reporting.

## Test/Sample and Test Programs

>     test_tools_example.cpp
>     test_tools_fail1.cpp
>     test_tools_fail2.cpp
>     test_tools_fail3.cpp
>     test_tools_fail4.cpp

## Header Implementation Option

The Boost Execution Tools main() function and Boost Test Tools supplied cpp_main() functions may be stored in an object library and linked into the final program just like any other supplied function. If this is not convenient, they can be compiled directly into the user's program by defining a macro:

>     #define BOOST_INCLUDE_MAIN
>     #include <boost/test/test_tools.hpp>

Although this usage is unusual, it is quite useful in some test environments where it may be difficult or undesirable to link to an object library.

## Rationale

How should a test program report errors?

Displaying an error message is an obvious possibility:

>     if ( something_bad_detected )
>       std::cout << "something bad has been detected" << std::endl;

But that requires inspection of the program's output after each run to determine if an error occurred. Since test programs are often run as part of a regression test suite, human inspection of output to detect error messages is too time consuming and unreliable. Test frameworks like GNU/expect can do the inspections automatically, but are overly complex for simple testing.

A better simple way to report errors is for the test program to return EXIT_SUCCESS (normally 0) if the test program completes satisfactorily, and EXIT_FAILURE if an error is detected. This allows a simple regression test script to automatically and unambiguous detect success or failure. Further appropriate actions such as creating an HTML table or emailing an alert can be taken by the script, and can be modified as desired without having to change the actual C++ test programs.

A testing protocol based on a policy of test programs returning EXIT_SUCCESS or EXIT_FAILURE does not require any supporting tools; the C++ language and standard library are sufficient. The programmer must remember, however, to catch all exceptions and convert them to program exits with non-zero return codes. The programmer must also remember to not use the standard library assert() macro for test code, because on some systems it results in undesirable side effects like a message requiring manual intervention.

The Test Library automates those tasks, yet can be ignored by programmers who prefer to implement the zero return testing protocol themselves.

## Design

The Boost Test Library Design document describes the relationship between the Boost Test Library components.

© Copyright Beman Dawes 2000

Revised 28 Feb 2001