# Boost Build System

## Synopsis

Boost.Build is a system for large project software construction built on Boost Jam, a descendant of "Perforce Jam", an open-source make replacement[1]. Key features are:

- A simple target description language
- Build with your choice (or multiple) toolsets from a single command invocation
- Build your choice of basic variants (e.g. debug, release, profile...) and subvariant modifications (e.g. inlining off) from a single command invocation
- ``Feature Normalization'' allows target configurations to be described independently from the toolset used
- Modular toolset description files allow build instructions for different toolsets to be described independently
- Multiple subproject support
- Automatic building of subproject dependencies

Here are some of the design criteria that led to these features.

## Table of Contents

- Synopsis
- Getting Started
  - Installing Jam
  - Initiating a Build
  - Setting Jam Variables
  - An Example Jamfile
  - Support Files
- Basic Design and Terminology
  - Projects and Subprojects
  - Targets
  - Features and Properties
  - Build Variants
  - Subvariants
  - Dependent Targets
- Usage
  - The Command-Line
    - User Targets
    - Global Variables
  - Subproject Jamfiles
    - The `subproject` Rule
    - Describing Main Targets
    - Example
  - Feature Descriptions
  - Variant Descriptions
  - Toolset Description Files
    - Example
- Internals
  - Jam Fundamentals
  - Core Jam Extensions
    - Command-line and Environment Variable Quoting
    - Jambase Replacement
    - Rule Indirection
    - Argument Lists
    - Module Support
      - Declaration
      - Local Variables
      - Local Rules
      - The `RULENAMES` rule
      - The `IMPORT` rule
      - The `EXPORT` rule
      - The `CALLER_MODULE` rule
    - Local for Loop Variables
    - While Loops
    - Negative Indexing
    - Target Binding Detection
    - Return Code Inversion
    - Ignoring Return Codes
    - The `SUBST` Rule
    - The `JAM_VERSION` global variable
    - Debugging Support
      - The BACKTRACE rule
      - Profiling
      - Parser Debugging
      - Dependency Graph Output
  - Target Names
  - Global Variables
- Design Criteria
  - Assumptions
  - Requirements
- Footnotes

## Getting Started

### Installing Boost Jam

- The Boost Jam sources are located in the `tools/build/jam_src` subdirectory of the Boost installation.
- The Jam README contains basic installation instructions.
- If you are installing on Windows, the make process may prompt you to set some environment variables, and stop. Don't be alarmed; just follow the instructions and start over. Please keep in mind that thses variable settings are case-sensitive. The variable settings necessary for bootstrapping Jam are not needed once it has been built.
- Note that the supplied Makefile may require editing for your platform; see the Jam README for details. The Makefile is used for bootstrapping Jam; it builds Jam into a subdirectory called `bin.`platform.

### Initiating a Build

The easiest way to get going is to set the `BOOST_ROOT` environment variable to point at the Boost installation directory, though you can also set `BOOST_ROOT` on the command-line, using `-s...`. You can use the `TOOLS` variable to indicate which toolset(s) to build with, and the `BUILD` variable to describe how you want things built. In most cases it should be sufficient to invoke Jam with no variable settings. The following examples all assume that `BOOST_ROOT` has been set in the environment.

Here are some sample Boost Jam invocations:

Command Line(s) | Effects
--- | ---
    jam -sTOOLS=gcc my_target
| default (debug) `BUILD` of `my_target`with GCC
    jam -fallyourbase-path -sTOOLS="msvc gcc"
| default-build `all` with msvc and gcc
    set TOOLS=msvc
    jam
| Set an NT environment variable to always build with MSVC default-build `all`.
    jam -sBUILD=release
| release build `all` with default `TOOLS`:
    jam -sBUILD="debug release"
| debug and release build `all`.

### Setting Jam Variables

The "`-s`" options in the command lines above are passing variable settings to the build system. There are actually three ways to do that:

- Jam picks up variable settings from your environment by default, so you can set them there:

  >     > BUILD="debug release"  # assuming Unix
  >     > export BUILD
  >     > jam ...

  This approach can be OK for quick-and-dirty tests, but environment variable settings tend to be unstable and non-uniform across users and machines, so it's best not to rely on the environment much.
- Explicitly on the command-line, with the "`-s`" option.
- Directly in Jam code. A project's Jamrules file is a convenient place to make global settings.

### An Example Jamfile

Here is an example of a simple subproject Jamfile. In this example, it is assumed that the user has set `BOOST_ROOT`, either as an environment variable, on the command-line or in the project's `Jamrules` file:

>     subproject foo/bar/baz ; # path to here from project root

>     # A static library called 'baz'
>     lib baz : baz1.cpp baz2.cpp # C++ sources
>               parser/src/baz4.ll # Lex->C++ sources
>               parser/src/baz5.yy  # Yacc->C++ sources
>             : <include>$(BOOST_PARENT_DIRECTORY)  # Put boost in #include path
>         ;

>     # An executable called 'test'
>     exe test : <lib>baz # use the 'baz' library
>                baz_test.cpp   # C++ source
>              : <include>$(BOOST_ROOT)
>         ;

That's it! The build system takes care of the rest. If the you want to be able to build all subprojects from the project root directory, you can add a Jamfile at the root:

>     project-root ; # declare this to be the project root directory
>     # Read subproject Jamfiles
>     subinclude foo/bar/baz foo/bar/... ;
>     subinclude a/b/c ... ; # more subincludes

### Support Files

To use the build system, the following must be located in your project's root directory, or in a directory specified in the `BOOST_BUILD_INSTALLATION` variable. It is usually convenient to specify the `BOOST_BUILD_INSTALLATION` in your project's Jamrules file. The Boost Jamrules file shows an example.

Filename(s) | Meaning
--- | ---
toolset`-tools.jam` | Feature-to-command-line mapping for toolset.
`features.jam` | Abstract toolset feature descriptions.
`boost-base.jam` | Boost build system-specific rule definitions.
`unit-tests.jam` | Unit tests and assertions for boost Jam code.

The `boost-base.jam` file is temporary, and will eventually be compiled into our Jam executable.

## Basic Design and Terminology

This section gives an overview of the way that the system works, outlining the system's capabilities and overall design. It also introduces the terminology and concepts neccessary to understand the sections on writing Jamfiles and command-line invocations.

### Projects and Subprojects

A project is a source directory tree containing at least one `Jamfile`. The root directory of the project is known as the project root. The root directory of a project may contain a `Jamrules` file, which contains project-specific Jam code. If the `Jamrules` file is not present when Jam is invoked, a warning will be issued.

Subdirectories containing `Jamfile`s are called subproject directories. Each such `Jamfile` describes a subproject.

The build system installation directory is a directory containing Jam files describing compilers and build variants. The installation directory can be specified explicitly by setting the variable `BOOST_BUILD_INSTALLATION`. If the installation directory is not specified, it is the same as the project root, and `BOOST_BUILD_INSTALLATION` is set to refer to that directory.

### Targets

Each `Jamfile` describes one or more main targets.

Each main target is an abstract description of one or more built targets which are expressions of the corresponding main target under particular compilers and build variants. Intermediate files such as `.o`/`.obj` files generated by compiling `.cpp` files as a consequence of building a main target are also referred to as built targets. The term build directory tree refers to the location of built target files.

- By default, the build directory tree is overlaid with the project directory tree, with targets generated into a subtree rooted at the `bin` subdirectory of each subproject directory (the name of this directory can be customized by changing the `BIN_DIRECTORY` variable.
- If the variable `ALL_LOCATE_TARGET` is set, it specifies an alternate build directory tree whose structure mirrors that of the project. In this case, built targets of a subproject are generated into the corresponding directory of the build directory tree.

For each main target, there is a corresponding location in the build directory tree known as the target's build root, where all intermediate and final targets resulting from that main target are located.

### Features and Properties

A feature is a normalized (toolset-independent) description of an individual build parameter, such as whether inlining is enabled. Each feature usually corresponds to a command-line option of one or more build tools. Features come in three varieties:

1. Simple features can take on any of several predetermined values. For example, the feature `optimization` might take one of the values `off`, `speed`, or `space`. Simple features have a default value. The key aspect of simple features is that they are assumed to affect link compatibility: object files generated with different values for a simple feature are generated into a separate directories, and (with a few exceptions) main targets generated with different values won't be linked together.
2. Free features can either be single-valued, as above, or may take on any number of user-specified values simultaneously. For example, the `define` feature for a release build might have the values `NDEBUG` and `BOOST_RELEASE_BUILD`. Free features are assumed not to affect link compatibility.
3. Path features are free features whose values describe paths which may be relative to the subproject (such as linked libraries or `#include` search directories). The build system treats the values of these features specially to ensure that they are interpreted relative to the subproject directory regardless of the directory where Jam was invoked.
4. Dependency features are path features whose values describe a dependency of built targets. For example, an external library might be specified with a dependency-feature: if the library is updated, the target will be updated also. The `<library-file>` feature works this way [2].

A feature-value pair is known as a build property, or simply property. The prefixes simple, free, path, and dependency apply to properties in an analogous way to features.

### Build Variants

A build variant, or simply variant is a named set of build properties describing how targets should be built. Typically you'll want at least two separate variants: one for debugging, and one for your release code.

Built targets for distinct build variants and toolsets are generated in separate parts of the build directory tree, known as the variant directories. For example, a (sub)project with main targets `foo` and `bar`, compiled with both GCC and KAI for `debug` and `release` variants might generate the following structure (target directories in bold).

>      bin
>      +-foo  <--- foo's build root
>      | +-gcc
>      | | +-debug
>      | | `-release
>      | `-kai
>      |   +-debug
>      |   `-release
>      `-bar  <--- bar's build root
>        +-gcc
>        | +-debug
>        | `-release
>        `-kai
>          +-debug
>          `-release

The properties constituting a variant may differ according to toolset, so `debug` may mean a slightly different set of properties for two different compilers.

### Subvariants

When a target is built with simple properties that don't exactly match those specified in a build variant, the non-matching features are called subvariant features and the target is located in a subvariant directory beneath the directory of the base variant. This can occur for two reasons:

1. Some features are only relevant to certain compilers. When relevant simple features have no value specified in the build variant, a value must be chosen. Even when the default value is used, the target is generated into a subvariant directory. For example, the `runtime-link` feature may be unspecified in the `debug` variant, but relevant to MSVC. In that case, a fragment of the target tree might look like:

   >      bin
   >      +-foo  <--- foo's build root
   >      | +-msvc
   >      | | +-debug
   >      . . . `-runtime-link-dynamic
   >      . . .

   Because the default value of `runtime-link` is `dynamic`, when the `debug` variant is requested, the `runtime-link-dynamic` subvariant of foo is built.
2. It is possible to request (either on the command-line, or as part of a main target description) that particular subvariants be built. For example, it may be desirable to generate builds that link to the runtime both statically and dynamically. In that case, both subvariant directories in the example above would be generated:

   >      bin
   >      +-foo  <--- foo's build root
   >      | +-msvc
   >      | | +-debug
   >      . . . +-runtime-link-dynamic
   >      . . . `-runtime-link-static
   >      . . .

In no case will targets be built directly into `bin/foo/msvc/debug`, since the `debug` variant doesn't include the `runtime-link` feature, which is relevant to MSVC.

When a subvariant includes multiple subvariant features, targets are built into a subvariant directory whose path is determined by concatenating the properties sorted in order of their feature names. For example, the borland compiler, which uses different libraries depending on whether the target is a console or GUI program, might create the following structure for a DLL:

>      bin
>      +-foo  <--- foo's build root
>      | +-msvc
>      | | +-debug
>      | | | +-runtime-link-dynamic
>      | | | | +-user-interface-console
>      | | | | `-user-interface-gui
>      . . . `-runtime-link-static
>      . . .   +-user-interface-console
>      . . .   `-user-interface-gui

Any configuration of properties for which a target is built, whether base variant or subvariant, is known as a build configuration, or simply a build.

### Dependent Targets

When a main target depends on the product of a second main target (as when an executable depends on and links to a static library), each build configuration of the dependent target is depends on the corresponding build of the dependency. Because only simple features participate in build identity, the dependent and dependency targets may have completely different free features. This puts the onus on the user for ensuring link-compatibility when certain free properties are used. For example, when `assert()` is used in header files, the preprocessor symbol `NDEBUG` can impact link-compatibility of separate compilation units. This danger can be minimized by encapsulating such feature differences inside of build variants.

## Usage

This section describes how to start a build from the command-line and how to write project and subproject Jamfiles. It also describes the other files written in the Jam language: build-tool specification files, feature descriptions files.

### The Command Line

This section describes in detail how the build system can be invoked.

#### User Targets

The Jam command line ends with an optional list of target names; if no target names are supplied, the built-in pseudotarget `all` is built. In a large project, naming targets can be dicey because of collisions. Jam uses a mechanism called grist to distinguish targets that would otherwise have the same name. Fortunately, you won't often have to supply grist at the command-line. When you declare a main target, a Jam pseudotarget of the same name is created which depends on all of the subvariants requested for your invocation of the build system. For example, if your subproject declares:

>     exe my_target : my_source1.cpp my_source2.c ;

and you invoke Jam with `-sBUILD="debug release" my_target`, you will build both the debug and release versions of `my_target`.

These simple, ungristed names are called user targets, and are only available for the subproject where Jam is invoked. That way, builds from the top level (which may include many Jamfiles through the subinclude rule) and builds of library dependencies (which may live in other subprojects), don't collide. If it is neccessary to refer more explicitly to a particular target from the command-line, you will have to add ``grist''. Please see this section for a more complete description of how to name particular targets in a build.

#### Global Variables

This is a partial list of global variables that can be set on the command-line. Of course you are free to write your own Jam rules which interpret other variables from the command-line. This list just details some of the variables used by the build system itself. Note also that if you don't like the default values you can override them in your project's `Jamrules` file.

Variable | Default | Example | Notes
--- | --- | --- | ---
`TOOLS` | Platform-dependent | `-sTOOLS="gcc msvc"` | build with gcc and msvc
`-sTOOLS=gcc` | build with gcc
`BUILD` | `debug` | `-sBUILD=release` | build the `release` variant
`-sBUILD="debug release"` | build both `debug` and `release` variants
`-sBUILD="<optimization>speed"` | build a subvariant of the default variant (`debug`) with optimization for speed.
`-sBUILD="debug release <runtime-link>static/dynamic"` | build subvariants of the debug and release variants that link to the runtime both statically and dynamically.
`ALL_LOCATE_TARGET` | empty | `-sALL_LOCATE_TARGET=~/build` | Generate all build results in the `build` subdirectory of the user's home directory (Unix).

### SubProject Jamfiles

This section describes how to write a Jamfile for a subproject.

#### The `subproject` rule

A subproject's Jamfile begins with an invocation of the `subproject` rule that specifies the subproject's location relative to the top of the project tree:

>     subproject path-from-top ;

The `subproject` rule tells the build system where to place built targets from the subproject in case `ALL_LOCATE_TARGET` is used to specify the build directory tree. If there is a Jamfile in the project root directory, you should use the `project-root` rule instead:

>     project-root ;

#### Describing Main Targets

A main target is described using the following syntax:

>     target-type name : sources
>         [ : requirements [ : default-BUILD ] ] ;

- target-type may be one of `exe`, `lib`, or `dll`. These are actually names of Jam rules. Additional main target rules are possible; see `status/Jamfile` or `libs/python/build/Jamfile` for examples.
- name specifies the name of the main target
- sources is a list of paths to source files and dependency targets. A dependency target path is preceded by `<lib>`, and the final path component specifies the name of a main target in a Jamfile located in the directory given by the inital path components. Paths may be absolute or relative.
- requirements specifies the build properties intrinsic to the target. Requirements are given as sets of optionally-qualified build properties:

  >     [[<compiler>]<variant>]<feature>value

  `<compiler>` and `<variant>`, if supplied, can be used to restrict the applicability of the requirement. Either one may be replaced by `<*>`, which is the same as ommitting it.

  The system checks that simple feature requirements are not violated by explicit subvariant build requests, and will issue a warning otherwise. Free features specified as requirements are simply added to each corresponding build configuration.

- default-BUILD specifies the configurations that should be built if the `BUILD` variable is not otherwise specified. Any elements not beginning with ```<`...`>`'' refer to build variants. Other elements use the same syntax as the requirements described above, except that multiple values may be specified for a simple feature by separating them with a slash, forming (qualified) multi-valued properties:

  >     [[<compiler>]<variant>]<feature>value1[/value2...]

  When multiple values are specified, it causes all the implied configurations to be built by default. It is also possible to prevent any default builds from occuring on this target by using `<suppress>true` . This suppresses any local targets, either implicit or explicit, from building. But, this does not prevent implied targets as required by a dependency by another target to this one from being built. This is usefull, for example, for defining a set of libraries generically and having them built only when another target like an exe is built. Such use might look like:

  >     lib basic : basic.cpp : : <suppress>true ;

  >     exe test : test.cpp <lib>basic ;

  With that the `basic` library will only be built when the `test` executable is built, and only the variations required by the executable will be built.

NOTE: for simple features in both requirements and default-BUILD, more-specific qualification overrides less-specific.

#### Example

This artificially complex example shows how an executable called "foo" might be described in a Jamfile. The executable is composed of the sources `./foo.cpp` and `./src/bar.cpp` (specified relative to the directory in which the Jamfile resides), and the built target which results from building the target `baz` as described in `../bazlib/Jamfile`.

>     exe foo : foo.cpp src/bar.cpp <lib>../bazlib/baz
>         ## Requirements ##
>         : <include>../bazlib/include
>           <define>BUILDING_FOO=1
>           <release><define>FOO_RELEASE
>           <msvc><*><define>FOO_MSVC
>           <msvc><release><define>FOO_MSVC_RELEASE
>           <gcc><*><optimization>off
>           <gcc><release><optimization>space
>           <threading>multi
>           <sysinclude>/usr/local/foolib/include

>         ## default-BUILD ##
>         : debug release
>           <debug><runtime-link>static/dynamic
>         ;

The requirements section:

- Adds `../bazlib/include` to the `#include` path
- Sets the preprocessor symbol `BUILDING_FOO` to `1`
- In the `release` builds, `#define`s `FOO_RELEASE`.
- When built with MSVC, `#define`s `FOO_MSVC`.
- In `release` variants built with MSVC, `#define`s `FOO_MSVC_RELEASE`.
- Most builds under GCC have optimization turned off, but...
- ...GCC `release` builds require optimization for space.
- Requires multithread support on compilers where it's relevant.
- Adds `/usr/local/foolib/include` to the `#include <*>` path

The default-BUILD section:

- specifies that `debug` and `release` base variants are built by default.
- on compilers where the feature is relevant, requests both statically- and dynamically-linked subvariants of the debug variant.

### Feature Descriptions

Features are described by stating the feature type (simple features are specified with "`feature`"), followed by the feature name. An optional second argument can be used to list the permissible values of the feature. Examples can be found in features.jam.

### Variant Descriptions

Variants are described with the following syntax:

>     variant name : [<toolset-name>]<feature>value... ;

The `variant` rule specifies the list of properties comprising a variant. Properties may be optionally qualified with a toolset name, which specifies that the property applies only to that toolset. One or more parent variants may be specified to inherit the properties from those parent(s). For inherited properties precedence is given on a left to right order, making the immediate properties override those in the parent(s). This can be used to great effect for describing global properties that are shared amongst various variants, and therefore targets. For example:

>     variant my-globals : <rtti>off ;

>     variant my-debug : my-globals debug ;

>     variant mym-release : my-globals releaase ;

More examples can be found in features.jam.

### Toolset Description Files

Toolset descriptions are located in the project's root directory, or a directory specified by `BOOST_BUILD_INSTALLATION`, which may be set in a `Jamfile` or the project's `Jamrules` file. Each file is called toolset-name`-tools.jam`, where toolset-name is the name of the toolset. The toolset description file has two main jobs:

1. redefine the following rules:
   - `Link-action` - links an executable from objects and libraries
   - `Archive-action` - links a static library from object files
   - `C++-action` - compiles a 'C++' file into an object file
   - `Cc-action` - compiles a 'C' file into an object file
   These rules should simply invoke the action part of a rule whose name is uniquely defined for the toolset. For example,

   >     rule C++-action
   >     {
   >         msvc-C++-action $(<) : $(>) ;
   >     }

   >     actions msvc-C++-action
   >     {
   >         cl -nologo -GX -c -U$(UNDEFS) -D$(DEFINES) $(CFLAGS) $(C++FLAGS) -I$(HDRS) -I$(STDHDRS) -Fo$(<) -Tp$(>)
   >     }

   Note that `Link-action` may require special care: on platforms where the global variable `gEXPORT_SUFFIX(DLL)` is defined (e.g. Windows), the first argument may have two elements when linking a shared library. The first is the shared library target, and the second is the import library target, with suffix given by `$(gEXPORT_SUFFIX(DLL))`. It will always have a third argument which is either ```EXE`'' or ```DLL`''. This can be used to dispatch to different actions for linking DLLs and EXEs if neccessary, but usually it will be easier to take advantage of the special `<target-type>` feature, which will have the same value using the `flags` rule described below.

2. Translate build settings given in the global `gBUILD_PROPERTIES` variable into something that can be used by the toolset. The build system provides the `flags` rule to help translate build properties into elements of global variables which are later attached to targets so that they can affect the build actions. The `flags` rule is used as follows:

   >     flags toolset variable condition [: value...]

   The parameters are:
   - toolset - the name of the toolset
   - variable - the name of a global variable which can be used to carry information to a command-line
   - condition - one one or more elements in the following forms:
     1. a property-set of the form: `<feature>value`[`/<feature> value`...]
     2. `<feature>`
   - values - anything

   Semantics only affect targets built with the specified toolset, and depend on the target's build configuration:

   1. if any specified property-set is a subset of the target's build properties, the values specified in `$(3)` will be appended once to variable.
   2. The value of each specified feature that participates in the target's build properaties is appended to variable. In either case, the variable will be set "on" the target so it may be used in the build actions.

#### Example

The description of the `flags` rule above is actually more complicated than it sounds. For example, the following line might be used to specify how optimization can be turned off for MSVC:

>     flags msvc CFLAGS <optimization>off : /Od ;

It says that the string `/Od` should be added to the global `CFLAGS` variable whenever a build configuration includes the property `<optimization>off`.

Similarly, in the following example,

>     flags msvc CFLAGS <runtime-build>release/<runtime-link>dynamic : /MD ;

we add `/MD` to the CFLAGS variable when both of the specified conditions are satisfied. We could grab all of the values of the free feature `<include>` in the `HDRS` variable as follows:

>     flags msvc HDRS <include> ;

The use of these variables should be apparent from the declaration of `actions msvc-C++-action` in the previous section.

## Internals

### Jam Fundamentals

This section is derived from the official Jam documentation and from my experience using it and reading the Jambase rules. I repeat the information here mostly because it is essential to understanding and using Jam, but is not consolidated in a single place. Some of it is missing from the official documentation altogether. I hope it will be useful to anyone wishing to become familiar with Jam and the Boost build system.

- Jam ``rules'' are actually simple procedural entities. Think of them as functions. Arguments are separated by colons.

- A Jam target is an abstract entity identified by an arbitrary string. The build-in `DEPENDS` rule creates a link in the dependency graph between the named targets.

- Note that the documentation for the built-in `INCLUDES` rule is incorrect: `INCLUDES targets1 : targets2` causes everything that depends on a member of targets1 to depend on all members of targets2. It does this in an odd way, by tacking targets2 onto a special tail section in the dependency list of everything in targets1. It seems to be OK to create circular dependencies this way; in fact, it appears to be the ``right thing to do'' when a single build action produces both targets1 and targets2.

- When a rule is invoked, if there are `actions` declared with the same name as the rule, the `actions` are added to the updating actions for the target identified by the rule's first argument. It is actually possible to invoke an undeclared rule if corresponding actions are declared: the rule is treated as empty.

- Targets (other than `NOTFILE` targets) are associated with paths in the file system through a process called binding. Binding is a process of searching for a file with the same name as the target (sans grist), based on the settings of the target-specific `SEARCH` and `LOCATE` variables.

- In addition to local and global variables, jam allows you to set a variable `on` a target. Target-specific variable values can usually not be read, and take effect only in the following contexts:

  - In updating `actions`, variable values are first looked up `on` the target named by the first argument (the target being updated). Because Jam builds its entire dependency tree before executing `actions`, Jam rules make target-specific variable settings as a way of supplying parameters to the corresponding `actions`.
  - Binding is controlled entirely by the target-specific setting of the `SEARCH` and `LOCATE` variables, as described here.
  - In the special rule used for header file scanning, variable values are first looked up `on` the target named by the rule's first argument (the source file being scanned).

- The ``bound value'' of a variable is the path associated with the target named by the variable. In build `actions`, the first two arguments are automatically replaced with their bound values. Target-specific variables can be selectively replaced by their bound values using the bind action modifier.

- Note that the term ``binding'' as used in the Jam documentation indicates a phase of processing that includes three sub-phases: binding (yes!), update determination, and header file scanning. The repetition of the term ``binding'' can lead to some confusion. In particular, the Modifying Binding section in the Jam documentation should probably be titled ``Modifying Update Determination''.

- ``Grist'' is just a string prefix of the form `<`characters`>`. It is used in Jam to create unique target names based on simpler names. For example, the file name ```test.exe`'' may be used by targets in separate subprojects, or for the debug and release variants of the ``same'' abstract target. Each distinct target bound to a file called ```test.exe`'' has its own unique grist prefix. The Boost build system also takes full advantage of Jam's ability to divide strings on grist boundaries, sometimes concatenating multiple gristed elements at the beginning of a string. Grist is used instead of identifying targets with absolute paths for two reasons:

  1. The location of targets cannot always be derived solely from what the user puts in a Jamfile, but sometimes depends also on the binding process. Some mechanism to distinctly identify targets with the same name is still needed.
  2. Grist allows us to use a uniform abstract identifier for each built target, regardless of target file location (as allowed by setting `ALL_LOCATE_TARGET`.
  When grist is extracted from a name with `$(`var`:G)`, the result includes the leading and trailing angle brackets. When grist is added to a name with `$(`var`:G=`expr`)`, existing grist is first stripped. Then, if expr is non-empty, leading `<`s and trailing `>`s are added if neccessary to form an expression of the form `<`expr2`>`; `<`expr2`>` is then prepended.

- When Jam is invoked it imports all environment variable settings into corresponding Jam variables, followed by all command-line (`-s...`) variable settings. Variables whose name ends in `PATH`, `Path`, or `path` are split into string lists on OS-specific path-list separator boundaries (e.g. "`:`" for Unix and "`;`" for Windows). All other variables are split on space ("`` ") boundaries. Boost Jam modifies that behavior by allowing variables to be quoted.

- A variable whose value is an empty list or which consists entirely of empty strings has a negative logical value. Thus, for example, code like the following allows a sensible non-empty default which can easily be overridden by the user:

  >     MESSAGE ?= starting jam... ;
  >     if $(MESSAGE) { ECHO The message is: $(MESSAGE) ; }

  If the user wants a specific message, he invokes jam with `"-sMESSAGE=`message text`"`. If he wants no message, he invokes jam with `-sMESSAGE=` and nothing at all is printed.

Please also read The Jam language reference for the additional details, and the Jam release notes for a brief description of recent, but fundamental changes to the Jam language without which you will probably not understand any of the build system code. In particular, note that the `return` statement does not affect control flow.

### Core Jam Extensions

A number of enhancements have been made to the core language of Classic Jam. These changes were aimed primarily at making it easier to manage the complexity of a large system such as Boost.Build.

> #### Command-line and Environment Variable Quoting

> Classic Jam had an odd behavior with respect to command-line variable ( `-s...`) and environment variable settings which made it impossible to define an arbitrary variable with spaces in the value. Boost Jam remedies that by treating all such settings as a single string if they are surrounded by double-quotes. Uses of this feature can look interesting, since shells require quotes to keep characters separated by whitespace from being treated as separate arguments:

> >     jam -sMSVCNT="\"\"C:\Program Files\Microsoft Visual C++\VC98\"\"" ...

> The outer quote is for the shell. The middle quote is for Jam, to tell it to take everything within those quotes literally, and the inner quotes are for the shell again when paths are passed as arguments to build actions. Under NT, it looks a lot more sane to use environment variables before invoking jam when you have to do this sort of quoting:

> >     set MSVCNT=""C:\Program Files\Microsoft Visual C++\VC98\""

> #### Jambase Replacement

> New logic has been added to allow the built-in Jambase to be replaced without recompiling Jam or adding command-line arguments. The user can control the location of the build system by setting any of the `JAMBASE`, `BOOST_ROOT`, or `BOOST_BUILD_PATH` environment variables (the settings of these variables can also be overridden on the command-line using the `-sVARIABLE=`... option).

> The process is controlled by variables (in decreasing precedence):

> - If `JAMBASE` is set, it specifies the path to the Jambase replacement. Non-rooted paths are computed relative to the directory of Jam's invocation.
> - Otherwise, if `BOOST_BUILD_PATH` or `BOOST_ROOT` is set, the build system filename is `boost-build.jam`.
> - If the build system filename does not contain a path specification, the build system file is searched for on `$(BOOST_BUILD_PATH)`, then at `$(BOOST_ROOT)/tools/build`.
> - If `BOOST_BUILD_PATH` was not set, it will be set to `$(BOOST_ROOT)/tools/build`.
> - If neither `JAMBASE`, `BOOST_ROOT`, nor `BOOST_BUILD_PATH` is set, we use the built-in Jambase (nearly identical to the FTJam Jambase) and load the user's Jamfile. Perforce Jam has this behavior, and it is used for building Jam itself. Thus, when you rebuild Jam, these variables should be unset.

> The rationale for this behavior is as follows:

> - The Jam executable should allow the Jambase to be overridden to implement other build systems without the user having any knowledge of Boost, thus the `JAMBASE` variable.
> - `BOOST_BUILD_PATH` is designed to be used to find all modules used by the build system, so that users and system administrators may non-intrusively add modules to the system.
> - Many Boost users already have `BOOST_ROOT` set. If a user doesn't want to explicitly set up `BOOST_BUILD_PATH`, `BOOST_ROOT` will supply reasonable behavior.

> #### Rule Indirection

> Boost Jam allows you to call a rule whose name is held in a variable or computed as the result of an expression:

> >     x = foo ;
> >     rule foobar { ECHO foobar ; }   # a trivial rule
> >     $(x)bar ;                       # invokes foobar

> Furthermore, if the first expression expands to more than one list item, everything after the first item becomes part of the first argument. This allows a crude form of argument binding:

> >     # return the elements of sequence for which predicate returns non-nil
> >     rule filter ( sequence * : predicate + )
> >     {
> >         local result ;
> >         for local x in $(sequence)
> >         {
> >             if [ $(predicate) $(x) ] { result += $(x); }
> >         }
> >         return $(result);
> >     }

> >     # true iff x == y
> >     rule equal ( x y )
> >     {
> >         if $(x) = $(y) { return true; }
> >     }

> >     # bind 3 to the first argument of equal
> >     ECHO [ filter 1 2 3 4 5 4 3 : equal 3 ] ; # prints "3 3"

> #### Argument lists

> You can now describe the arguments accepted by a rule, and refer to them by name within the rule. For example, the following prints ``I'm sorry, Dave'' to the console:

> >     rule report ( pronoun index ? : state : names + )
> >     {
> >         local he.suffix she.suffix it.suffix = s ;
> >         local I.suffix = m ;
> >         local they.suffix you.suffix = re ;

> >         ECHO $(pronoun)'$($(pronoun).suffix) $(state), $(names[$(index)]) ;
> >     }
> >     report I 2 : sorry : Joe Dave Pete ;

> Each name in a list of formal arguments (separated by ```:`'' in the rule declaration) is bound to a single element of the corresponding actual argument unless followed by one of these modifiers:

> Symbol | Semantics of preceding symbol
> --- | ---
> `?` | optional
> `*` | Bind to zero or more unbound elements of the actual argument. When ```*`'' appears where an argument name is expected, any number of additional arguments are accepted. This feature can be used to implement "varargs" rules.
> `+` | Bind to one or more unbound elements of the actual argument.

> The actual and formal arguments are checked for inconsistencies, which cause Jam to exit with an error code:

> >     ### argument error
> >     # rule report ( pronoun index ?  : state  : names + )
> >     # called with: ( I 2 foo  : sorry  : Joe Dave Pete )
> >     # extra argument foo

> >     ### argument error
> >     # rule report ( pronoun index ?  : state  : names + )
> >     # called with: ( I 2  : sorry )
> >     # missing argument names

> If you omit the list of formal arguments, all checking is bypassed as in ``classic'' Jam. Argument lists drastically improve the reliability and readability of your rules, however, and are strongly recommended for any new Jam code you write.

> #### Module Support

> Boost Jam introduces support for modules, which provide some rudimentary namespace protection for rules and variables. A new keyword, ```module`'' was also introduced. The features described in this section are primitives, meaning that they are meant to provide the operations needed to write Jam rules which provide a more elegant module interface.

> > ##### Declaration

> > `module` expression`{` ...`}`

> > Code within the `{` ...`}` executes within the module named by evaluating expression. Rule definitions can be found in the module's own namespace, and in the namespace of the global module as module-name`.`rule-name, so within a module, other rules in that module may always be invoked without qualification:

> > >     module my_module
> > >     {
> > >         rule salute ( x ) { ECHO $(x), world ; }
> > >         rule greet ( ) { salute hello ; }
> > >         greet ;
> > >     }
> > >     my_module.salute goodbye ;

> > When an invoked rule is not found in the current module's namespace, it is looked up in the namespace of the global module, so qualified calls work across modules:

> > >     module your_module
> > >     {
> > >         rule bedtime ( ) { my_module.salute goodnight ; }
> > >     }

> > ##### Local Variables

> > `module local` expression`;`
> > - or -
> > `module local` expression`=` expression2`;`

> > The variables named by expression are given a distinct value in the module, and can be manipulated by code executing in the module without affecting variable bindings seen by other modules. If the assignment form is used, expression2 is assigned to the variables when the declaration is executed. For example:

> > >     module M {
> > >       module local x = a b c ;

> > >       rule f ( )
> > >       {
> > >         {
> > >           local x = 1 2 3 ; # temp override for M's x
> > >           N.g ;             # call into module N, below
> > >         }
> > >         ECHO $(x) ;         # prints "a b c"
> > >       }
> > >     }

> > >     module N {
> > >       rule g ( )
> > >       {
> > >         x = foo bar baz ; # sets global x
> > >         M.h ;             # call back into M, below
> > >       }
> > >     }

> > >     module M {
> > >       rule h ( )
> > >       {
> > >         ECHO $(x) ;      # prints "1 2 3"
> > >       }
> > >     }

> > >     M.f ;
> > >     ECHO $(x) ;          # prints "foo bar baz"

> > The only way to access another module's local variables is through a rule defined in that module:

> > >     module M {
> > >       rule get ( names * )
> > >       {
> > >         return $($(names)) ;
> > >       }
> > >     }
> > >     ECHO [ M.get x ] ;  # prints "a b c"

> > ##### Local Rules

> > > `local rule` rulename...

> > The rule is declared locally to the current module. It is not entered in the global module with qualification, and its name will not appear in the result of

> > > `[ RULENAMES` module-name`]`.

> > ##### The `RULENAMES` Rule

> > >     rule RULENAMES ( module ? )

> > Returns a list of the names of all non-local rules in the given module. If `module` is ommitted, the names of all non-local rules in the global module are returned.

> > ##### The `IMPORT` Rule

> > `IMPORT` allows rule name aliasing across modules:

> > >     rule IMPORT (  source_module ? : source_rules *
> > >                     : target_module ? : target_rules * )

> > The `IMPORT` rule copies rules from the `source_module` into the `target_module` as `local` rules. If either `source_module` or `target_module` is not supplied, it refers to the global module. `source_rules` specifies which rules from the `source_module` to import; `TARGET_RULES` specifies the names to give those rules in `target_module`. If `source_rules` contains a name which doesn't correspond to a rule in `source_module`, or if it contains a different number of items than `target_rules`, an error is issued. For example,

> > >     # import m1.rule1 into m2 as local rule m1-rule1.
> > >     IMPORT m1 : rule1 : m2 : m1-rule1 ;

> > >     # import all non-local rules from m1 into m2
> > >     IMPORT m1 : [ RULENAMES m1 ] : m2 : [ RULENAMES m1 ] ;

> > ##### The `EXPORT` Rule

> > `EXPORT` allows rule name aliasing across modules:

> > >     rule EXPORT ( module ? : rules * )

> > The `EXPORT` rule marks `rules` from the `source_module` as non-local (and thus exportable). If an element of `rules` does not name a rule in `module`, an error is issued. For example,

> > >     module X {
> > >       local rule r { ECHO X.r ; }
> > >     }
> > >     IMPORT X : r : : r ; # error - r is local in X
> > >     EXPORT X : r ;
> > >     IMPORT X : r : : r ; # OK.

> > ##### The `CALLER_MODULE` Rule

> > >     rule CALLER_MODULE ( levels ? )

> > `CALLER_MODULE` returns the name of the module scope enclosing the call to its caller (if levels is supplied, it is interpreted as an integer number of additional levels of call stack to traverse to locate the module). If the scope belongs to the global module, or if no such module exists, returns the empty list. For example, the following prints "{Y} {X}":

> > >     module X {
> > >         rule get-caller { return [ CALLER_MODULE ] ; }
> > >         rule get-caller's-caller { return [ CALLER_MODULE 1 ] ; }

> > >         rule call-Y { return Y.call-X2 ; }

> > >     }
> > >     module Y {
> > >         rule call-X { return X.get-caller ; }
> > >         rule call-X2 { return X.get-caller's-caller ; }
> > >     }

> > >     callers = [ X.get-caller ] [ Y.call-X ] [ X.call-Y ] ;
> > >     ECHO {$(callers)} ;

> #### Local For Loop Variables

> Boost Jam allows you to declare a local `for` loop control variable right in the loop:

> >     x = 1 2 3 ;
> >     y = 4 5 6 ;
> >     for local y in $(x)
> >     {
> >         ECHO $(y) ; # prints "1", "2", or "3"
> >     }
> >     ECHO $(y) ;     # prints "4 5 6"

> #### While Loops

> In classic Jam, some constructs are only possible using recursion:

> >     # returns the part of $(list) following the first occurrence of $(symbol)
> >     rule after-symbol ( symbol : list * )
> >     {
> >         if ! $(list) || ( $(symbol) = $(list[1]) )
> >         {
> >             return $(list[2-]) ;
> >         }
> >         else
> >         {
> >             return [ after-symbol $(symbol) : $(list[2-]) ] ;
> >         }
> >     }

> The addition of `while` loops allows a simpler formulation for this and many other rules:

> >     rule after-symbol ( symbol : list * )
> >     {
> >         while $(list) && $(list[1]) != $(symbol)
> >         {
> >             list = $(list[2-]) ;
> >         }
> >         return $(list) ;
> >     }

> #### Negative Indexing

> Classic Jam supplies 1-based list indexing, and slicing on a closed (inclusive) range:

> >     x = 1 2 3 4 5 ;
> >     ECHO $(x[3]) ;   # prints "3"
> >     ECHO $(x[2-4]) ; # prints "2 3 4"
> >     ECHO $(x[2-]) ;  # prints "2 3 4 5"

> Boost Jam adds Python-style negative indexing to access locations relative to the end of the list.

> >     ECHO $(x[-1]) $(x[-3]) ; # prints "5 3"
> >     ECHO $(x[-3--1]) ;       # prints "3 4 5"
> >     ECHO $(x[-3-4]) ;        # prints "3 4"
> >     ECHO $(x[2--2]) ;        # prints "2 3 4"

> Consistency with the 1-based, inclusive indexing of Classic Jam and the use of ```-`'' as the range separator make this feature a bit clumsier than it would otherwise need to be, but it does work.

> #### Target Binding Detection

> Whenever a target is bound to a location in the filesystem, Boost Jam will look for a variable called `BINDRULE` (first ```on`'' the target being bound, then in the global module). If non-empty, `$(BINDRULE[1])` names a rule which is called with the name of the target and the path it is being bound to. The signature of the rule named by `$(BINDRULE[1])` should match the following:

> >     rule bind-rule ( target : path )

> This facility is useful for correct header file scanning, since many compilers will search for `#include`d files first in the directory containing the file doing the `#include` directive. `$(BINDRULE)` can be used to make a record of that directory.

> #### Return Code Inversion

> For handling targets whose build actions are expected to fail (e.g. when testing that assertions or compile-time type checkin work properly), Boost Jam supplies a `FAIL_EXPECTED` rule in the same style as `NOCARE`, et. al. During target updating, the return code of the build actions for arguments to `FAIL_EXPECTED` is inverted: if it fails, building of dependent targets continues as though it succeeded. If it succeeds, dependent targets are skipped.

> #### Ignoring Return Codes

> Perforce Jam suppplied a `NOCARE` rule which is typically used for header files to indicate that if they are not found, the dependent targets should be built anyway. Boost Jam extends `NOCARE` to apply to targets with build actions: if their build actions exit with a nonzero return code, dependent targets will still be built.

> #### The `SUBST` Rule

> The behavior of the `SUBST` rule for regular-expression matching and replacement (originally added in FTJam) has been modified:

> - One or more replacement patterns may be supplied. The new signature for `SUBST` is:

>   >     SUBST ( source pattern replacements + )

>   The return value is the concatenated results of applying each element of `replacements` in turn. For example, the following will print ```[x] (y) {z}`'':

>   >     ECHO [ SUBST xyz (.)(.)(.) [$1] ($2) {$3} ] ;

> - If there is no match, `SUBST` now returns an empty list. In FTJam, the original `source` string was returned, making it awkward to check whether a pattern was matched.

> - Compiled regular expressions are now internally cached, making it much faster to use `SUBST` multiple times with the same string.

> #### The `JAM_VERSION` global variable

> A predefined global variable with two elements indicates the version number of Boost Jam. Boost Jam versions start at `"03" "00"`. Earlier versions of Jam do not automatically define `JAM_VERSION`.

> #### Debugging Support

> ##### The BACKTRACE rule

> >     rule BACKTRACE ( )

> Returns a list of quadruples: filename line module rulename..., describing each shallower level of the call stack. This rule can be used to generate useful diagnostic messages from Jam rules.

> The `-d` command-line option admits new arguments:

> - `-d+10` - enables profiling of rule invocations. When Jam exits, it dumps all rules invoked, their gross and net times in platform-dependent units, and the number of times the rule was invoked.
> - `-d+11` - enables parser debugging, if Jam has been compiled with the "--debug" option to the parser generator named by $(YACC).
> - `-d+12` - enables dependency graph output . This feature was ``stolen'' from a version of Jam modified by Craig McPheeters.

### Target Names

In addition to user targets, which correspond directly to the names the user writes in her subproject Jamfile, several additional targets are generated, regardless of the directory from which Jam was invoked:

- A main target has all the same dependencies as a user target (i.e. building it updates all requested subvariants). Its name is the same except for the addition of `$(SOURCE_GRIST)`, which identifies the subproject. The identification looks like the names of the path components from the project root to the subproject, separated by exclamation points. Thus, if the project is rooted at `foo`, in the subproject at `foo/bar/baz` the target `my_target` is identified by `<bar!baz>my_target`.
- A subvariant target has additional grist identifying its main target and subvariant. This grist is joined to `$(SOURCE_GRIST)` with the platform's directory separator. Thus, on Unix, a subvariant target of `my_target` above might be identified as `<bar!baz/my_target/optimization-space/runtime-link-static>my_source.o`. Note that the part of the grist following the first slash, known as the subvariant id, also corresponds to a fragment of the subvariant directory path where the corresponding target is generated. Most built targets will be identified this way.

### Global Variables

This section describes some of the global variables used by the build system. Please note that some parts of the system (particularly those in `allyourbase.jam`) are heavily based on the Jambase file supplied with Jam, and as such do not follow the conventions described below.

Global variables used in the build system fall into three categories:

- Global variables intended to be set by the user on the command-line or in the environment use `ALL_UPPER_CASE` names.
- Internal global variables begin with a lower-case "g" and continue in upper-case: `gSOME_GLOBAL`
- Global variables of the form: `gBASE_NAME(`arguments`)`, where arguments is a comma-separated argument list, are used internally to achieve a kind of indirection by concatenating variable values:

  >         ECHO $(gFUBAR($(x),$(y))) ;

Please note that the build system commonly takes advantage of Jam's Dynamic Scoping feature (see the `local` command in the "Flow of Control" section below the link target) to temporarily "change" a global variable by declaring a `local` of the same name.

### Variables Associated with SubProject Identity

- `SUBDIR_TOKENS` - a list of the path elements relative to the project root of the current subproject.
- `SUBDIR` - the path from the invocation directory to the current subproject directory.

### Grist Variables

- `TARGET_GRIST` takes the form `subproject!id/target/toolset/variant/subvariant-path`.

## Design Criteria

### Assumptions

The requirements are driven by several basic assumptions:

- There is no single Boost developer or test facility with access to or knowledge of all the platforms and compilers Boost libraries are used with.
- Boost libraries are used across such a wide range of platforms and compilers that almost no other assumptions can be made.

### Requirements

This build system was designed to satisfy the following requirements:

- A developer adding a new library or test program must only have to add simple entries naming the source files to a text file, and not have to know anything about platform specific files. The developer should not have to supply header dependency information.
- There should be a very high likelihood of builds succeeding on all platforms if a build succeeds on any platform. In other words, a developer must not be required to have access to many platforms or compilers to ensure correct builds
- A user or developer adding support for a new platform or compiler should only have to add to a single file describing how to do the build for that platform or compiler, and shouldn't have to identify the files that will need to be built.
- The build should rely only on tools native to the platform and compiler, or supplied via the boost download.
- The details of how the build is done for a particular platform or compiler should be appropriate for that platform.
- It should be possible to build multiple variants (e.g. debug/release) of a single target.
- It should be possible to build multiple variants of multiple targets with multiple compilers from a single build command.
- The build tools must be able to handle Boost growth issues such as identified in Directory Structure proposals and discussion.
- Support for dynamic and static linking should be included.
- It should be relatively straightforward to add support for a new compiler. In most cases, no modification of files used to describe existing targets should be required.
- Support for compiler- and variant-specific configuration for each target
- It should be possible to build targets into a directory unrelated to the source directories (they may be read-only)

## Footnotes

[1] Boost Jam is actually descended directly from FTJam, which was itself a variant of Jam/MR. It is hoped that crucial features we rely on will eventually be incorportated back into the Jam/MR release.

[2] Note: right now, a dependency feature of a main target makes all resulting built targets dependent, including intermediate targets. That means that if an executable is dependent on an external library, and that library changes, all the sources comprising the executable will be recompiled as well. This behavior should probably be fixed.

© Copyright David Abrahams 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.

Revised 11 November, 2001