How Meson is tested

A build system is a very important part of the development workflow and people depend on it to work reliably. In order to achieve this we do a lot of testing on Meson. As this is mostly invisible to end users I thought I'd write some information about our testing setup and practices.

Perhaps the most unconventional thing is that Meson has no unit tests in the traditional sense of the word. The code consists of functions, classes and modules as you would expect but there are no test for these individual items. Instead all testing is done on full projects. The bulk of tests are what could traditionally be called integration tests. That is, we take an entire project that does one thing, such as compile and install a shared library, and then configures it, builds it, runs tests and runs install and checks that the output is correct.

There are also smaller scale tests which are called unit tests. They also configure a project but then inspect the result directly. As an example test might set up a project and build it, and then touch one file and verify that it triggers a rebuild. Some people might claim that these are not unit tests either or that they should instead be tested with mock classes and the like. This is a valid point to make, but this is the terminology we have converged unto.

Enter The (Testing) Matrix

Our testing matrix is big. Really big. At its core are the three main supported platforms, Linux, Windows and OSX. We also support BSD's and the like but we currently don't have CI machines for them.

On Windows our Appveyor setup tests VS2010, 2015 and 2017 and in addition mingw and Cygwin. Apart from Cygwin these all are tested on both 32 and 64 bits variants. Visual studio is tested both with the Ninja and Visual Studio project generator backends.

OSX is the simplest, it is tested only with the Ninja backend using both regular and unity builds.

Linux tests do a lot more. In addition running the basic tests (both in unity and regular modes) it also runs the entire test suite in a cross compilation setup.

All of these tests are only for the core code. Meson supports a large amount of frameworks and libraries, such as GLib, Qt and Doxygen, and many programming languages. Every one of these has an associated test or, usually, several. This means that running the full test suite on Debian requires installing all of these:

• Boost
• D, Rust, Java, Fortran, C# and Swift compilers
• Qt 5, WxWidgets, GTK+ 3
• Valgrind
• GNUstep
• Protocol Buffers
• Cython
• GTest and GMock

And a bunch of other packages as well. The CI Docker image that has all of these installed takes 2 gigabytes of space. On many distros all dependencies are not available so packagers have to disable tests. Having a build dependency on all these packages sometimes yields interesting problems. As an example the Rust dependency means that Meson depends on LLVM. Every now and then it breaks on s390x meaning that Meson and every package that uses it to build get flagged for removal from Debian.

Every merge proposal to Meson master is run through all of these tests and is eligible for merging only if they all pass. There are no exceptions to this rule. 

There are some downsides to this, the biggest being that every now and then Appveyor and/or Travis get clogged and getting the green light takes forever. We looked briefly into getting paid instances but for our usage the bill would be in the neighborhood of $300 per month. Given that you can buy your own hardware for that kind of money, this has not been seen as a worthwhile investment.