Dependencies with code generators got a lot smoother with Meson 0.46.0

Most dependencies are libraries. Almost all build systems can find dependency libraries from the system using e.g. pkg-config. Some can build dependencies from source. Some, like Meson, can do both and toggle between them transparently. Library dependencies might not be a fully solved problem but we as a community have a fairly good grasp on how to make them work.

However there are some dependencies where this is not enough. A fairly common case is to have a dependency that has some sort of a source code generator. Examples of this include Protocol Buffers, Qt's moc and glib-mkenums and other tools that come with Glib. The common solution is to look up these binaries from PATH. This works for dependencies that are already installed on the system but fails quite badly when the dependencies are built as subprojects. Bootstrapping is also a bit trickier because you may need to write custom code in the project that provides the executables.

Version 0.46.0 which shipped yesterday has new functionality that makes this use case noticeably simpler. In Meson you find the code generator scripts to run with the find_program command like this:

mkenums_exe = find_program('glib-mkenums')

This will find the executable from the system. However if you have built Glib as a subproject, then it can issue the following statements (this is not in Glib master yet AFAIK so it does not work, this is more of an illustrative example):

internal_mkenums_exe = <command to generate the mkenum script>
meson.override_find_program('glib-mkenums', internal_mkenums_exe)

After this issuing find_program('glib-mkenums') no longer goes to the system, but instead returns the internal program. Meson's internal helper modules have also been updated to always find the programs they use with find_program. This means that all projects using Glib functionality can be built without needing a system wide install of Glib. Even more importantly this requires zero changes in existing projects. It will just work out of the box. You can even use Glib helper code when building Glib itself.

This is especially convenient when you need a newer version of any dependency than your distro provides and especially on platforms such as Windows where "distro dependencies" do not exist.

As an example of what is possible, Nirbheek has managed to bootstrap GStreamer on Windows using nothing but Visual Studio, Python 3, Ninja and Meson. The main limitation currently is that the overriding executable may not be a build target (i.e. something you build from source with a compiler) because the result of find_program may be used during the configuration phase, before any source code compilation has taken place. We hope to remove this limitation in a future release.

Cookie purging the simple way

Getting rid of cookies (especially tracking and ad cookies) consistently is a good thing. However it turns out to be a bit tricky because you don't want to get rid of session cookies for sites you care about. Basically what you want to achieve is this:

1. Store all cookies as normal
2. Maintain a whitelist of servers that are allowed to store persistent cookies (usually for sites such as Github, Reddit, Twitter and the like)
3. At regular intervals (preferably every time the browser is closed), delete all cookies not whitelisted.

There are browser extensions to do this but they are often bizarrely complex and even those that aren't are inconvenient to use as they require installing plugins, clicking through menus and so on. Firefox should have builtin functionality to do this also, but I read through instructions online on how to do it and could not understand how you should set it up to get it to work.

Thus as an experiment I wrote a Python script to do this, it is available in this Github repo. Using it is simple:

1. Write a whitelist file consisting of one hostname per line. (all subdomains of the specified host are also permitted)
2. Shut down Firefox.
3. Run the script.
4. Start Firefox.

Comparing Meson with Bazel on a Raspberry Pi 3

In this experiment we compile Google's Abseil C++ libraries with Bazel and also with Meson as a simple comparison of how they behave.

Apple and orange warning!

Please do not use this text to exclaim that one of these build systems is better/more performant/etc than the other. That's not really what this is for. The two build systems build the code completely differently to different artifacts and with different targets. Consider this a more of a rough outline.

The Meson conversion

The code was converted with a simple script and then manually fixing some dependency declarations to make everything work. Due to complicated reasons there's no Git repo. Instead you can download the whole shebang as a zip file from this location. It's not against current trunk, but instead a random commit from some time ago when I started.

The original Bazel build does a bunch of complicated things with shared libraries and the like. The Meson one simply builds a static library for each of the Abseil modules. This is not particularly efficient but I just wanted to get something out without spending days replicating the build setup.

The build setup is complete enough to build all unit tests apart from ones that seem to require magic compiler flags, because they give compilation errors about missing timespec definitions.

Memory usage

When doing the actual compilation, Meson is not resident in memory. Only Ninja is, and it takes 2-5 MB of memory in total.

The Bazel master process takes roughly 90 MB when compiling. That is almost 10% of total system memory.

CPU utilization

Based on top/htop eyeballing, Ninja keeps all cores pegged almost all of the time. The time command reported CPU usage of 328%. It was necessary to manually specify -j 4 to Ninja (the default value is 6) because otherwise the system would hard freeze under load, most likely due to memory running out.

Weirdly Bazel had a really hard time keeping cores running. It was common to have 1-3 cores idle (that is, not even waiting for IO) during the build. It is not known what causes this. Perhaps a lot of time is spent doing the file copies and symlinks that Bazel needs for its hermetic builds. But even then, maximal usage of resources is one of Bazel's claimed strong points but in this particular case that does not seem to be happening. It is possible that the behaviour has been tuned to data centers with tens of cores and fast SSDs and because of that does not scale down to ARM processors with an SD card for storage.

Total compile time

Meson used 6 minutes whereas Bazel used 17 minutes. But note the text above! Do not use this as any sort of "real" perf measurement because the setups were different. That being said if you consider that Ninja gets up to 3x better CPU utilization, the numbers seem to be in the rough neighborhood as far as total CPU usage is concerned.

Ninja reports doing 190 build steps whereas Bazel reports a number on the order of 4-500. Many of these seem to deal with file copying and the like which the Meson setup does not do at all. Effort was not spent on examining what these steps do and how (or if) they could be replicated in Meson.

Which one is better/which one should I use/which subreddit should I post this to?

That's not what this post was about. Do your own tests and draw your own conclusions.