Managing dependencies with Meson + WrapDB

A recent blog post talked about how to build and manage dependencies with CMake and FetchContent. The example that they used was a simple GUI application using the SFML multimedia libraries and the Dear ImGui widget toolkit using the corresponding wrapper library. For comparison let's do the same with Meson.

The basics

The tl/dr version is that all you need is to install Meson, check out this repository and start the build. Meson will automatically download all dependencies as source, build them from scratch and create a single statically linked final executable. SFML's source archive contains a bunch of prebuilt dependencies for OpenAL and the like. Those are not used, everything is compiled from original sources. A total of 11 subprojects are used including things like libpng, Freetype and Flac.

They are downloaded from Meson's WrapDB dependency provider/package manager service. which combines upstream tarballs with user submitted Meson build definitions. The only exception is stb. It has no releases, instead it is expected to be used directly from Git. As WrapDB only provides actual releases, this dependency needs to be checked out from a custom Git repo. This is entirely transparent to the user, the only change is that the wrap file specifying where the dependency comes from points to a different place.

If you actually try to compile the code you might face some issues. It has only been properly tested on Windows. It will probably work on Linux and most definitely won't work on macOS. At the time of writing GNU's web mirror has an expired certificate so downloading Freetype's release tarball will fail. You can work around it by downloading it by hand and placing it in the subprojects/packagecache directory. The build of SFML might also fail as the code uses auto_ptr, which has been removed from some stdlibs. This has been fixed in master (but not in the way you might have expected) but the fix has not made it to a release yet.

What does it look like?

I would have added an inline image here, but for some reason Blogger's image uploader is broken and just fails (on two different OSs even). So here's an imgur link instead.

This picture shows the app running. To the left you can also see all the dependencies that were loaded during the build. It also tells you why people should do proper release tarballs rather than relying on Github's autogenerated ones. Since every project's files are named v1.0.0.zip, the risk of name collision is high.

What's the difference between the two?

CMake has a single flat project space (or at least that is how it is being used here) which is used like this:

FetchContent_Declare(

  sfml

  URL https://github.com/SFML/SFML/archive/refs/tags/2.5.1.zip

  URL_MD5 2c4438b3e5b2d81a6e626ecf72bf75be

)

add_subdirectory(sfml)

I.e. "download the given file, extract it in the current directory (whatever it may be) and enter it as if it was our own code". This is "easy" but problematic in that the subproject may change its parent project in interesting ways that usually lead to debugging and hair pulling.

In Meson every subproject is compiled in its own isolated sandbox. They can only communicate in specific, well defined and secured channels. This makes it easy to generate projects that can be built from source on Windows/macOS/Android/etc and which use system dependencies on Linux transparently. This equals less hassle for everyone involved.

There are other advantages as well. Meson provides a builtin option for determining whether a project should build its libraries shared or static. This option can be set on the command line per subproject. The sample application project is set up to build everything statically for convenience. However one of the dependencies, OpenAL, is LGPL, so for final distributions you'll probably need to build it as a shared library. This can be achieved with the following command:

meson configure -Dopenal-soft:default_library=shared

After this only the OpenAL dependency is built as a shared library whereas everything else is still static. As this is a builtin, no project needs to write their own options, flags and settings to select whether to build shared or static libraries. Better yet, no end user has to hunt around to discover whether the option to change is FOO_BUILD_SHARED, FOO_ENABLE_SHARED, FOO_SHARED_LIBS, SHARED_FOO, or something else.

Why all open source maintainers are jerks, the Drake equation hypothesis

Preface

This blog post is meant to be humorous. It is not a serious piece of scientifically rigorous research. In particular it is not aiming to justify bad behaviour or toxicity in any way, shape or form. Neither does it claim that this mechanism is the only source of negativity. If you think it is doing any of these things, then you are probably taking the whole thing too seriously and are reading into it meanings and implications that are not there. If it helps, you can think of the whole thing as part of a stand-up comedy routine.

Why is everybody a jerk?

It seems common knowledge that maintainers of major open source projects are rude. You have your linuses, lennarts, ulrichs, robs and so on. Why is that? What is it about project maintenance that brings out these supposed toxics? Why can't projects be manned by nice people? Surely that would be better.

Let's examine this mathematically. For that we need some definitions. For each project we have the following variables:

• N, the total number of users
• f_p, the fraction of users who would like to change the program to better match their needs
• f_r, the fraction of users with a problem who file a change request
• f_rej, the fraction of submitted change requests that get rejected
• f_i, the fraction of of people who consider rejections as attacks against their person
• f_c, the fraction of people who complain about their perceived injustices on public forums
• f_j, the fraction of complainers formulating their complaints as malice on the maintainer's part

Thus we have the following formula for the amount of "maintainer X is a jerk" comments on the net:

J = N * f_p * f_r * f_rej * f_i * f_c * f_j

Once J reaches some threshold, the whisper network takes over and the fact that someone is a jerk becomes "common knowledge".

The only two variables that a maintainer can reasonably control are N and f_rej (note that f_i can't ever be brought to zero, because some people are incredibly good at taking everything personally) Perceived jerkness can thus only be avoided either by having no users or accepting every single change request ever filed. Neither of these is a good long term strategy.

.C as a file extension for C++ is not portable

Some projects use .C as a file extension for C++ source code. This is ill-advised, because it is can't really be made to work automatically and reliably. Suppose we have a file source.C with the following contents:

class Foo {
public:
    int x;
};

Let's compile this with the default compiler on Linux:

$ cc -c -o /dev/null source.C

Note that that command is using the C compiler, not the C++ one. Still, the compiler will autodetect the type from the extension and compile it as C++. Now let's do the same thing using Visual Studio:

$ cl /nologo /c source.C
source.C(1): Error C2061 Syntax error: Identifier 'Foo'
<a bunch of other errors>

In this case Visual Studio has chosen to compile it as plain C. The defaults between these two compilers are the opposite and that leads to problems.

How to fix this?

The best solution is to change the file extension to an unambiguous one. The following is a simple ZSH snippet that does the renaming part:

for i in **/*.C; do git mv ${i} ${i:r}.cpp; done

Then you need to do the equivalent in your build files with search-and-replace.

If that is not possible, you need to use the /TP compiler switch with Visual Studio to make it compile the source as C++ rather than C. Note that if you use this argument on a target, then all files are built as C++, even the plain C ones. This is unreliable and can lead to weird bugs. Thus you should rename the files instead.

Is the space increase caused by static linking a problem?

Most recent programming languages want to link all of their dependencies statically rather than using shared libraries. This has many implications, but for now we'll only focus on one: executable size. It is generally accepted that executables created in this way are bigger than when static linking. The question is how much and whether it even mattesr. Proponents of static linking say the increase is irrelevant given current computers and gigabit networks. Opponents are of the, well, opposite opinion. Unfortunately there is very little real world measurements around for this.

Instead of arguing about hypotheticals, let's try to find some actual facts. Can we find a case where, within the last year or so, a major proponent of static linking has voluntarily switched to shared linking due to issues such as bandwidth savings. If such a case can be found, then it would indicate that, yes, the binary size increase caused by static linking is a real issue.

Android WebView, Chrome and the Trichrome library

Last year (?)  Android changed the way they provide both the Chrome browser and the System WebView app [1]. Originally both of them were fully isolated, but after the change both of them had a dependency on a new library called Trichrome, which is basically just a single shared library. According to news sites, the reasoning was this:

"Chrome is no longer used as a WebView implementation in Q+. We've moved to a new model for sharing common code between Chrome and WebView (called "Trichrome") which gives the same benefits of reduced download and install size while having fewer weird special cases and bugs."

Google has, for a long time, favored static linking. Yet, in this case, they have chosen to switch from static linking to shared linking on their flagship application on their flagship operating system. More importantly their reasons seem to be purely technical. This would indicate that shared linking does provide real world benefits compared to static linking.

[1] I originally became aware of this issue since this change broke updates on both of these apps and I had to fix it manually with apkmirror.

"Should we break the ABI" is the wrong question

The ongoing battle on breaking C++'s ABI seems to be gathering steam again. In a nutshell there are two sets of people in this debate. The first set wants to break ABI to gain performance and get rid of bugs, whereas the second set of people want to preserve the ABI to keep old programs working. Both sides have dug their heels in the ground and refuse to budge.

However debating whether the ABI should be broken or not is not really the issue. A more productive question would be "if we do the break, how do we keep both the old and new systems working at the same time during a transition period". That is the real issue. If you can create a good solution to this problem, then the original problem goes away because both sides get what they want. In other words, what you want to achieve is to be able to run a command like this:

prog_using_old_abi | prog_using_new_abi

and have it work transparently and reliably. It turns out that this is already possible. In fact many (most?) people are reading this blog post on a computer that already does exactly that.

Supporting multiple ABIs at the same time

On Debian-based systems this is called multi-arch support. It allows you to, for example, run 32 and 64 bit apps transparently on the same machine at the same time. IIRC it was even possible to upgrade a 32 bit OS install to 64 bits by first enabling the 64 bit arch and then disabling the 32 bit one. The basic gist of multiarch is that rather than installing libraries to /usr/lib, they get installed to something like /usr/lib/x86_64. The kernel and userspace tools know how to handle these two different binary types based on the metadata in ELF executables.

Given this we could define an entirely new processor type, let's call it x86_65, and add that as a new architecture. Since there is no existing code we can do arbitrary changes to the ABI and nothing breaks. Once that is done we can create the full toolchain, rebuild all OS packages with the new toolchain and install them. The old libraries remain and can be used to run all the old programs that could not be recompiled (for whatever reason). 

Eventually the old version can be thrown away. Things like old Steam games could still be run via something like Flatpak. Major corporations that have old programs they don't want to touch are the outstanding problem case. This just means that Red Hat and Suse can sell them extra-long term support for the old ABI userland + toolchain for an extra expensive price. This way those organizations who don't want to get with the times are the ones who end up paying for the stability and in return distro vendors get more money. This is good.

Simpler ABI tagging

Defining a whole virtual CPU just for this seems a bit heavy handed and would probably encounter a lot of resistance. It would be a lot smoother if there were a simpler way to version this ABI change. It turns out that there is. If you read the ELF specification, you will find that it has two fields for ABI versioning (and Wikipedia claims that the first one is usually left at zero). Using these fields the multiarch specification could be expanded to be a "multi-abi" spec. It would require a fair bit of work in various system components like the linker, RPM, Apt and the like to ensure the two different ABIs are never loaded in the same process. As an bonus you could do ABI breaking changes to libc at the same time (such as redefining intmax_t) There does not seem to be any immediate showstoppers, though, and the basic feasibility has already been proven by multiarch.

Obligatory Rust comparison

Rust does not have a stable ABI, in fact it is the exact opposite. Any compiler version is allowed to break the ABI in any way it wants to. This has been a constant source of friction for a long time and many people have tried to make Rust commit to some sort of a stable ABI. No-one has been successful. It is unlikely they will commit to distro-level ABI stability in the foreseeable future, possibly ever.

However if we could agree to steady and continuous ABI transitions like these every few years then that is something that they might agree to. If this happens then the end result would be beneficial to everyone involved. Paradoxically it would also mean that by having a well established incremental way to break ABI would lead to more ABI stability overall.