# Cross-extension-module dependencies

It is good programming practice to organize large projects as modules that interact with each other via well defined interfaces. With Boost.Python it is possible to reflect this organization at the C++ level at the Python level. This is, each logical C++ module can be organized as a separate Python extension module.

At first sight this might seem natural and straightforward. However, it is a fairly complex problem to establish cross-extension-module dependencies while maintaining the same ease of use Boost.Python provides for classes that are wrapped in the same extension module. To a large extent this complexity can be hidden from the author of a Boost.Python extension module, but not entirely.

## The recipe

Suppose there is an extension module that exposes certain instances of the C++ `std::vector` template library such that it can be used from Python in the following manner:

    import std_vector
    v = std_vector.double([1, 2, 3, 4])
    v.push_back(5)
    v.size()

Suppose the `std_vector` module is done well and reflects all C++ functions that are useful at the Python level, for all C++ built-in data types (`std_vector.int`, `std_vector.long`, etc.).

Suppose further that there is statistic module with a C++ class that has constructors or member functions that use or return a `std::vector`. For example:

    class xy {
      public:
        xy(const std::vector<double>& x, const std::vector<double>& y) : m_x(x), m_y(y) {}
        const std::vector<double>& x() const { return m_x; }
        const std::vector<double>& y() const { return m_y; }
        double correlation();
      private:
        std::vector<double> m_x;
        std::vector<double> m_y;
    }

What is more natural than reusing the `std_vector` extension module to expose these constructors or functions to Python?

Unfortunately, what seems natural needs a little work in both the `std_vector` and the `statistics` module.

In the `std_vector` extension module, `std::vector<double>` is exposed to Python in the usual way with the `class_builder<>` template. To also enable the automatic conversion of `std::vector<double>` function arguments or return values in other Boost.Python C++ modules, the converters that convert a `std::vector<double>` C++ object to a Python object and vice versa (i.e. the `to_python()` and `from_python()` template functions) have to be exported. For example:

      #include <boost/python/cross_module.hpp>
      //...
      class_builder<std::vector<double> > v_double(std_vector_module, "double");
      export_converters(v_double);

In the extension module that wraps `class xy` we can now import these converters with the `import_converters<>` template. For example:

      #include <boost/python/cross_module.hpp>
      //...
      import_converters<std::vector<double> > v_double_converters("std_vector", "double");

That is all. All the attributes that are defined for `std_vector.double` in the `std_vector` Boost.Python module will be available for the returned objects of `xy.x()` and `xy.y()`. Similarly, the constructor for `xy` will accept objects that were created by the `std_vector`module.

## Placement of `import_converters<>` template instantiations

`import_converts<>` can be viewed as a drop-in replacement for `class_wrapper<>`, and the recommendations for the placement of `class_wrapper<>` template instantiations also apply to to `import_converts<>`. In particular, it is important that an instantiation of `class_wrapper<>` is visible to any code which wraps a C++ function with a `T`, `T*`, const `T&`, etc. parameter or return value. Therefore you may want to group all `class_wrapper<>` and `import_converts<>` instantiations at the top of your module's init function, then `def()` the member functions later to avoid problems with inter-class dependencies.

## Non-copyable types

`export_converters()` instantiates C++ template functions that invoke the copy constructor of the wrapped type. For a type that is non-copyable this will result in compile-time error messages. In such a case, `export_converters_noncopyable()` can be used to export the converters that do not involve the copy constructor of the wrapped type. For example:

    class_builder<store> py_store(your_module, "store");
    export_converters_noncopyable(py_store);

The corresponding `import_converters<>` statement does not need any special attention:

    import_converters<store> py_store("noncopyable_export", "store");

## Python module search path

The `std_vector` and `statistics` modules can now be used in the following way:

    import std_vector
    import statistics
    x = std_vector.double([1, 2, 3, 4])
    y = std_vector.double([2, 4, 6, 8])
    xy = statistics.xy(x, y)
    xy.correlation()

In this example it is clear that Python has to be able to find both the `std_vector` and the `statistics` extension module. In other words, both extension modules need to be in the Python module search path (`sys.path`).

The situation is not always this obvious. Suppose the `statistics` module has a `random()` function that returns a vector of random numbers with a given length:

    import statistics
    x = statistics.random(5)
    y = statistics.random(5)
    xy = statistics.xy(x, y)
    xy.correlation()

A naive user will not easily anticipate that the `std_vector` module is used to pass the `x` and `y` vectors around. If the `std_vector` module is in the Python module search path, this form of ignorance is of no harm. On the contrary, we are glad that we do not have to bother the user with details like this.

If the `std_vector` module is not in the Python module search path, a Python exception will be raised:

    Traceback (innermost last):
      File "foo.py", line 2, in ?
        x = statistics.random(5)
    ImportError: No module named std_vector

As is the case with any system of a non-trivial complexity, it is important that the setup is consistent and complete.

## Two-way module dependencies

Boost.Python supports two-way module dependencies. This is best illustrated by a simple example.

Suppose there is a module `ivect` that implements vectors of integers, and a similar module `dvect` that implements vectors of doubles. We want to be able do convert an integer vector to a double vector and vice versa. For example:

    import ivect
    iv = ivect.ivect((1,2,3,4,5))
    dv = iv.as_dvect()

The last expression will implicitly import the `dvect` module in order to enable the conversion of the C++ representation of `dvect` to a Python object. The analogous is possible for a `dvect`:

    import dvect
    dv = dvect.dvect((1,2,3,4,5))
    iv = dv.as_ivect()

Now the `ivect` module is imported implicitly.

Note that the two-way dependencies are possible because the dependencies are resolved only when needed. This is, the initialization of the `ivect` module does not rely on the `dvect` module, and vice versa. Only if `as_dvect()` or `as_ivect()` is actually invoked will the corresponding module be implicitly imported. This also means that, for example, the `dvect` module does not have to be available at all if `as_dvect()` is never used.

## Clarification of compile-time and link-time dependencies

Boost.Python's support for resolving cross-module dependencies at runtime does not imply that compile-time dependencies are eliminated. For example, the statistics extension module in the example above will need to `#include <vector>`. This is immediately obvious from the definition of `class xy`.

If a library is wrapped that consists of both header files and compiled components (e.g. `libdvect.a`, `dvect.lib`, etc.), both the Boost.Python extension module with the `export_converters()` statement and the module with the `import_converters<>` statement need to be linked against the object library. Ideally one would build a shared library (e.g. `libdvect.so`, `dvect.dll`, etc.). However, this introduces the issue of having to configure the search path for the dynamic loading correctly. For small libraries it is therefore often more convenient to ignore the fact that the object files are loaded into memory more than once.

## Summary of motivation for cross-module support

The main purpose of Boost.Python's cross-module support is to allow for a modular system layout. With this support it is straightforward to reflect C++ code organization at the Python level. Without the cross-module support, a multi-purpose module like `std_vector` would be impractical because the entire wrapper code would somehow have to be duplicated in all extension modules that use it, making them harder to maintain and harder to build.

Another motivation for the cross-module support is that two extension modules that wrap the same class cannot both be imported into Python. For example, if there are two modules `A` and `B` that both wrap a given `class X`, this will work:

    import A
    x = A.X()

This will also work:

    import B
    x = B.X()

However, this will fail:

    import A
    import B
    python: /net/cci/rwgk/boost/boost/python/detail/extension_class.hpp:866:
    static void boost::python::detail::class_registry<X>::register_class(boost::python::detail::extension_class_base *):
    Assertion `static_class_object == 0' failed.
    Abort

A good solution is to wrap `class X` only once. Depending on the situation, this could be done by module `A` or `B`, or an additional small extension module that only wraps and exports `class X`.

Finally, there can be important psychological or political reasons for using the cross-module support. If a group of classes is lumped together with many others in a huge module, the authors will have difficulties in being identified with their work. The situation is much more transparent if the work is represented by a module with a recognizable name. This is not just a question of strong egos, but also of getting credit and funding.

## Why not use `export_converters()` universally?

There is some overhead associated with the Boost.Python cross-module support. Depending on the platform, the size of the code generated by `export_converters()` is roughly 10%-20% of that generated by `class_builder<>`. For a large extension module with many wrapped classes, this could mean a significant difference. Therefore the general recommendation is to use `export_converters()` only for classes that are likely to be used as function arguments or return values in other modules.

© Copyright Ralf W. Grosse-Kunstleve 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.

Updated: April 2001