Managing the build definitions of a big project with many subprojects and interdependencies

Last week the news broke that Boost is switching from their own build system to CMake. This made me finally look properly into how Boost is built and what lessons we can learn from it. The results turned out to be quite interesting.

For those interested in diving into Boost's code note that the source layout in Git repos is different from what it is in the release tarballs. The latter has a sort of a "preinstalled header" directory with all public headers whereas they are inside each individual repository in Git. There also seem to be two different sets of build definitions, one for each.

Creating a sample project

My first idea was to convert a subset of Boost into Meson for a direct comparison. I spent a lot of time looking at the Jamfiles and could not understand a single thing about them. So instead I created a demonstration project called Liftoff, which can be downloaded from Github. The project had the following requirements:

• support many standalone subprojects
• subprojects can depend on other subprojects
• shared dependencies are built only once, every project using it gets the same instance
• subprojects can be built either as shared or static libraries or used in a header only mode
• can build either all projects or only one + all its dependencies
• any dependency can also be obtained from the system if it is available
• monorepo layout, but support splitting it up into many individual repos if desired

The libraries

The project consists of four independent subprojects:

• lo_test, a simple unit testing framework
• lo_adder, a helper module for adding integers, depends on lt_test
• lo_strings, a helper module for manipulating strings, has no dependencies
• lo_shuttle, an application to launch shuttles, depends on all other modules

Note how both lo_adder and lo_shuttle depend on lo_test. Each subproject comes with a header and unit tests, some come with a dependency library as well.

The dependency bit

The core idea behind Meson's dependency system is that projects can declare dependency objects which specify how the dependency should be used (sort of like a Meson-internal pkg-config file). This is how it looks like for the string library:

lo_strings_dep = declare_dependency(link_with : string_lib,

  include_directories : include_directories('.'),

)

Other projects can then request this dependency object and use it to build their targets like this:

string_dep = dependency('lo_strings', fallback : ['lo_strings', 'lo_strings_dep'])

This is Meson nomenclature for "try to find the dependency from the system and if not found use the one in the given subproject". This dependency object can then be used in build targets and the build system takes care of the rest.

Building it

The build command from the command line is this:

meson build

ninja -C build test

This builds and runs all tests. Once you have it built, here are things to try:

• toggle between shared and static libraries with mesonconf -Ddefault_library=shared [or static]
• note how the test library is built only once, even though it is used by two different subprojects
• do a mesonconf -Dmodule=lo_strings and build, note that no other subproject is built anymore
• do a mesonconf -Dmodule=lo_adder and build, note that lo_test is built automatically, because it is a direct dependency of lo_adder

"Header only" dependencies

Some projects want to ship header only libraries but to also make it possible to build a helper library, usually to cut down on build times. This can be done but it is usually not pretty. You need to write "implementation header files" and do magic preprocessor incantations to ensure things are built in proper locations. We could replicate all of that in Meson if we wanted to, after all it's only grunt work. But we're not going to do that.

Instead we are going to do something fancier.

The main problem here is that traditionally there has been no way to tell that a dependency should also come with some source files that should be compiled in the dependent target. However in Meson this is supported. The lo_strings subproject can be set up to build in this way with the following command:

mesonconf build -Dlo_strings:header_only=true

When the project is built after this, the lo_strings project is not built, instead its source files are put inside the dependent targets and built there. Note that the build definition files for the dependent targets do not change at all. They are identical regardless of where your dependency comes from or how it should be built. Also switching between how things should be built does not require changing the build definition files, it can be toggled from "the outside".

How much space do the build definitions take in total?

66 lines.