# A testing system for Boost.Build

Introduction and examples
  Changing the working directory
  Examining the working directory and changing it
Reference documentation
  Method __init__
  Method `set_tree`
  Method `write`
  Method `copy`
  Method `touch`
  Method `run_build_system`
  Method `read`
  Methods for declaring expectations
  Methods for ignoring changes
  Helper class `List`

## Introduction and examples

The testing system for Boost.Build is derived from the `TestCmd` Python module used for testing the Scons build tools. The changes are very few, and I hope that some of them may be adopted by `TestCmd` in future.

Unfortunately, there are not documentation for `TestCmd`, apart from the comments in code, so the most essential base methods will be described here.

The basic steps of testing a build system behaviour are:

1. Setting the initial working directory state
2. Running the build system and checking:
   1. generated output,
   2. changes made to the working directory,
   3. new content of the working directory.
3. Adding, removing or touching files, or changing their content and then repeating the previous step, until satisfied.

Prior to using the test system, make sure you have checked out the Boost CVS and have successfully build Jam -- which means that `bjam` executable is available.

For example, suppose that you've placed under `tools/build/test` a file `test1.py` and a subdirectory `test1` with files `foo.cpp`, `Jamfile` and `Jamrules`. A definition of function `main` is the only content of `foo.cpp`, `Jamfile` builds an executable `foo` from `foo.cpp` and `Jamrules` is empty. Finally, `test1.py` contains:

    from BoostBuild import Tester, List

    # Create a temporary working directory
    t = Tester()

    # Make content of the working directory equal to the content of the 'test1' directory.
    t.set_tree('test1')

    # Invoke 'bjam -sTOOLS=borland' and record which changes were made
    t.run_build_system("-sTOOLS=borland")
    # First, create a list of three pathnames
    file_list = List("bin/foo/borland/debug/runtime-link-dynamic/foo") * List(".exe .obj")
    # Second, assert that those files were added as result of the last build system invocation.
    t.expect_addition(file_list)

    # Invoke the build system once again
    t.run_build_system("clean")
    # Check if the files added previously were removed.
    t.expect_removal(t.mul("bin/foo/borland/debug/runtime-link-dynamic/foo", [".exe", ".obj", ".tds"]))

This is all needed for a minimal test (and to find a bug in the borland toolset!). Running the `test1.py` gives the following output:

    File bin/foo/borland/debug/runtime-link-dynamic/foo.tds not removed as expected
    FAILED test of D:\MyDocu~1\Work\build\boost-build\boost-build -d0
            at line 144 of TestBoostBuild.py (expect_removal)
            from line 12 of test1.py

Overview of the most important methods of class `TestBoostBuild` follows.

### Changing the working directory

The class `TestBoostBuild` creates a temporary directory in its constructor and changes to that directory. It can be modified by calling these methods:

- `set_tree` -- sets the content of the working directory to be equal to the content of the specified directory.
- `write` -- sets the content of file in a working directory
- `touch` -- changes the modification times of a file

### Examining the working directory and changing it

The method `read`, inherited from the `TestCmd` class, can be used to read any file in the working directory and check its content. `TestBoostBuild` adds another method for tracking changes. Whenever build system is run (via `run_build_system`), the state of working dir before and after running is recorded. In addition, difference between the two states -- i.e. lists of files that were added, removed, modified or touched -- is stored in two member variables, `tree_difference` and `unexpected_difference`.

After than, the test author may specify that some change is expected, for example, by calling `expect_addition("foo")`. This call will check if the file was indeed added, and if so, will remove its name from the list of added files in `unexpected_difference`. Likewise, it's possible to specify that some changes are not interesting, for example a call `ignore("*.obj")` will just remove every files with ".obj" extension from `unexpected_difference`.

When test has finished with expectations and ignoring, the member `unexpected_difference` will contain the list of all changes not yet accounted for. It is possible to assure that this list is empty by calling `expect_nothing_more` member function.

## Reference documentation

The test system is composed of class `Tester`, derived form `TestCmd.TestCmd`, and helper class `List`. The methods of `Tester`, and the class `List` are described below.

The documentation frequently refer to filename. In all cases, files are specified in unix style: a sequence of components, separated by "/". This is true on all platforms. In some contexts, a list of files is allowed. In that case any object with sequence interface is allowed.

### Method `__init__(self, arguments='', executable='bjam')`

Effects:

1. Remembers the current working directory in member `original_workdir`.
2. Determines the location of executable (`bjam` by default) and build system files, assuming that the current directory is `tools/build/test`. Formulates jam invocation command, which will include explicit setting for `BOOST_BUILD_PATH` variable and arguments passed to this methods, if any. This command will be used by subsequent invocation of `run_build_system`. Finally, initializes the base class.
3. Changes current working dir to the temporary working directory created by the base constructor.

### Method `set_tree(self, tree_location)`

Effects:

Replaces the content of the current working directory with the content of directory at `tree_location`. If `tree_location` is not absolute pathname, it will be treated as relative to `self.original_workdir`. This methods also explicitly makes the copied files writeable.

### Method `write(self, name, content)`

Effects:

Writes the specified content to the file given by `name` under the temporary working directory. If the file already exists, it is overwritten. Any required directories are automatically created.

### Method `copy(self, src, dst)`

Effects:

Equvivalent to `self.write(self.read(src), dst)`.

### Method `touch(self, names)`

Effects:

Sets the access and modification times for all files in `names` to the current time. All the elements in `names` should be relative paths.

### Method `run_build_system(self, subdir='', extra_args='', stdout=None, stderr='', status=0, **kw)`

Effects:

1. Stores the state of the working directory in `self.previous_tree`.
2. Changes to `subdir`, if it is specified. If it is not absolute path, it is relative to the working dir.
3. Invokes the `bjam` executable, passing `extra_args` to it. The binary should be located under `<test_invocation_dir>/../jam_src/bin.<platform>`. This is to make sure tests use the version of jam build from CVS.
4. Compares the stdout, stderr and exit status of build system invocation with values to appropriate parameters, if they are not `None`. If any difference is found, the test fails.

5. Stores the new state of the working directory in `self.tree`. Computes the difference between previous and current trees and store them in variables `self.tree_difference` and `self.unexpected_difference`.

   Both variables are instances of class `tree.Trees_different`, which have four attributes: `added_files`, `removed_files`, `modified_files` and `touched_files`. Each is a list of strings.

### Method `read(self, name)`

Effects:

Read the specified file and returns it content. Raises an exception is the file is absent.

### Methods for declaring expectations

Accordingly to the number of changes kinds that are detected, there are four methods that specify that test author expects a specific change to occur. They check `self.unexpected_difference`, and if the change is present there, it is removed. Otherwise, test fails.

Each method accepts a list of names. Those names are always unix-style, even on other systems.

Note: The `List` helper class might be useful to create lists of names.

Note: The file content can be examined using `TestCmd.read` function.

The members are:

- expect_addition
- expect_removal
- expect_modification
- expect_touch
- expect_nothing

There's also a member `expect_nothing_more`, which checks that all the changes are either expected or ignored, in other words that `unexpected_difference` is empty by now.

### Methods for ignoring changes

There are five methods which ignore changes made to the working tree. They silently remove elements from `self.unexpected_difference`, and don't generate error if element is not found. They accept shell style wildcard.

The following methods correspond to four kinds of changes:

- ignore_addition(self, wildcard)
- ignore_removal(self, wildcard)
- ignore_modification(self, wildcard)
- ignore_touch(self, wilcard)

The method `ignore(self, wildcard)` ignores all the changes made to files that match a wildcard.

### Helper class `List`

The class has sequence interface and two additional methods.

#### Method `__init__(self, string)`

Effects: Splits the string on unescaped spaces and tabs. The split components can further be retrieved using standard sequence access.

#### Method `__mul__(self, other)`

Effects: Returns an `List` instance, which elements are all possible concatenations of two string, first of which is from `self`, and second of which is from `other`.

The class also defines `__str__` and `__repr__` methods. Finally, there's `__coerce__` method which allows to convert strings to instances of `List`.

Example:

        l = "a b" * List("c d")
        for e in l:
            print e

will output

        ac
        ad
        bc
        bd

Last modified: May 5, 2002

© Copyright Vladimir Prus 2002. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided ``as is'' without express or implied warranty, and with no claim as to its suitability for any purpose.