There is more to running an open source project than writing code. In fact most of all work has to do with something else. This places additional requirements to project maintainers that are often not talked about. In this post we'll briefly go over nine distinct phases each with a different hat one might have to wear. These can be split into two stages based on the lifetime and popularity of the project.
A gathering of development thoughts of Jussi Pakkanen. Some of you may know him as the creator of the Meson build system.
Monday, November 16, 2020
The Nine Phases of an Open Source Project Maintainer
Saturday, November 7, 2020
Proposal for target-private directories for compilers
One of the greatest strengths of the classical C compiler model is that all compile jobs are fully isolated. This means that they can be run perfectly in parallel. This same feature is also one of its greatest weaknesses. There are no ways for individual compile jobs to communicate with each other even if they wanted to. This could be useful for things like caching. As an example a compiler might transparently create "precompiled headers" of sorts during one compilation and use them in other compilations if needed. This might also be useful for languages that require scanning steps before building such as Fortran and C++ using modules.
This is not idle speculation. Clang's thinLTO does use caching to speed up incremental builds. Since there is no existing standard for this, they did the usual thing and created a new compiler flag for specifying the location of the cache directory. Or, to be precise, they created four of them:
- gold (as of LLVM 4.0): -Wl,-plugin-opt,cache-dir=/path/to/cache
- ld64 (support in clang 3.9 and Xcode 8): -Wl,-cache_path_lto,/path/to/cache
- ELF lld (as of LLVM 5.0): -Wl,--thinlto-cache-dir=/path/to/cache
- COFF lld-link (as of LLVM 6.0): /lldltocache:/path/to/cache
For one option this is tedious but for many it becomes completely unworkable. Clearly something better is needed.
The basic idea: each build target gets its own private directory
Basically what one should be able to do is this:
gcc -c -o foo/bar.o bar.c -fprivate-dir=some_path
The private directory would have the following properties:
- The build system guarantees that it is set to the same value for all compilations of a single target (executable, shared library, static library, etc)
- Every build target gets its own unique private directory
- The contents of the directory may persist over successive invocations (i.e. its contents may be deleted at any time, but most of the time won't be)
- The compiler can create anything it wants in the private dir but should also tolerate other usages (typically you'd also want to put the target's object files in this dir)
- The contents of the private dir are transitory, they have no backwards or forwards compatibility guarantees. Any compiler update would invalidate all files.
Work needed for tooling
Monday, November 2, 2020
You wanted Boost via Meson subprojects? You got it! (sorta)
In the previous blog post we saw a way to build SDL transparently as a Meson subproject. In the discussion that followed I got a question on whether you could consume Boost in the same way. This is an interesting question, because Boost is a, let's say, challenging dependency. It is very big and set up in an unusual way. As an example I would estimate that the single fact that they don't ship Pkg-Config files has cost Meson developers tens of hours of unnecessary troubleshooting. Having something simpler and more reliable would be welcome.
To test this out I created an MVP program that uses Boost's flat map from the container library and then added dependencies until it worked. The actual code can be downloaded here (tested on Linux, VS and Mac). The main program's basic build definition is as simple as the SDL program's was:
executable('boosttest', 'boosttest.cpp',
dependencies: [boost_container_dep])
The Boost container dep is specified in the container library's build file:
dependencies: [...])
As this is a header-only library, the only thing it needs to do is to expose the header dir. The dependencies keyword argument lists all the other dependencies that are needed to build code that uses the container library. These are move, assert, static_assert, intrusive, core and config. Their build files are almost identical to this one. No code changes were needed. The total LoC of meson.build files for this entire setup is 42. Which is apt.
Making it better
The main reason for this low line count is the fact that the Meson build definition do a lot less than the original ones. They do a lot are highly configurable, which might also explain why Boost's conversion to CMake has taken several years and is still ongoing. A lot of that effort is taken by things like documentation, but the actual build is also more complicated as it provides for more stuff. Here are two examples and an outline of how they would be implemented in Meson.
Built libraries
Some Boost libraries are header-only, others require some code to be built and linked against. Suppose we have a header-only dependency. Its dependency object would be defined like this:
Converting that to contain a library component would be done like this:
Basically you build the library and then tell users to link to that. This is pretty much what the SDL build definitions did. The library can be shared or static, they both work the same way.
Compiler flags needed for using the dependency
Some of the libraries seem to require that users of the library must specify some compiler command line arguments. These might or might not be the same ones that are used to build the library components themselves. This is natively supported.
How much work would it be to convert all of Boost?
Friday, October 30, 2020
How to build dependencies as Meson subprojects using SDL as an example
Today we released version 0.56.0 of the Meson build system. This is an especially important release as it marks the 10 000th commit since the start of the project. A huge thank you to everyone who has contributed their time and effort, this project would not exist without all of you. However in this post we are not going to talk about that, those interested can find further details in the release notes. Instead we are going to be talking about how to build your dependencies from source on every platform without needing anything other than Meson.
Last month I had a lightning talk at CppCon about this way of managing dependencies:
Since then there have been many improvements to the workflow for a smoother experience. To demonstrate this I upgraded the sample program to use SDL Mixer and SDL Image instead of relying on plain SDL. The code is available in this Github repo (only tested on Windows because I ran out of time to do proper multiplatform testing) The core of the build definition is this:
sdl2_image_dep = dependency('sdl2_image')
sdl2_mixer_dep = dependency('sdl2_mixer')
executable('sdltestapp', 'main.cpp',
dependencies : [sdl2_image_dep, sdl2_mixer_dep, sdl2_dep],
win_subsystem: 'windows')
This has always worked on Linux and other platforms that provide system dependencies via Pkg-Config. As of the latest release of Meson and newest code from WrapDB, this works transparently on all platforms. Basically for every dependency you want to build yourself, you need a wrap file, like this:
The contents consist mostly of download links, hashes and build meta info. Upstream sources are not checked in your repo (unless you want to) so it remains small. The actual files are in the repository linked above. When you start building and the project needs some dependency, Meson will use info in wrap files to download, patch and build the dependencies as subprojects as needed. This is what it looks like when configuring on Windows using MSVC without any external dependency providers.
Then you can build it. Here I'm using Visual Studio Code because why not.
The end result runs directly as a native Windows application (text annotations not part of the original program) using SDL's Direct3D accelerated rendering.
There are three different source files loaded using SDL Image: a png file, a jpg file and an LZW compressed tif file. The app is also playing sound effects in Ogg Vorbis and wav formats using SDL Mixer. Dependencies of dependencies work automatically as one would expect. All dependency libraries are statically linked as it is the easiest way to get things working under Windows (at least until your project gets too big).
If you want to try it yourself, here's what you need to do:
- Install Visual Studio, Meson and Ninja (not strictly required, you can use the VS backend instead if you wish)
- Open the VS x64 dev tools shell.
- Clone the repo and cd inside it.
- Run meson build
- Run ninja -C build
Contributing to the WrapDB
Submitting projects that already build with Meson
- Make sure your project gets all its dependencies via the dependency function rather than invoking subproject directly and provide a suitable dependency object via declare_dependency.
- Create an upstream.wrap file with the necessary info. See the documentation or, if you prefer examples, how it is done in libtiff.
- Request that your project be added to WrapDB as described in the documentation linked above.
Contributing projects that do not build with Meson
Monday, October 26, 2020
The Meson Manual: Good News, Bad News and Good News
Starting with good news, the Meson Manual has been updated to a third edition. In addition to the usual set of typo fixes, there is an entirely new chapter on converting projects from an existing build system to Meson. Not only are there tips and tricks on each part of the conversion, there is even guidance on how to get it done on projects that are too big to be converted in one go.
Unfortunately there are also bad news, which boils down to this graph.
This is the monthly sales amounts (in euros) since the beginning of this year. As you can tell it follows a typical exponential decay with a peak at the beginning and then a steady decline. Sales for this month are expected to be noticeably smaller than last month. Unfortunately keeping online sales for a product like this ongoing requires both time and money (to pay for the various services needed). Unfortunately the numbers show that this is not financially worthwhile.
Thus I must inform everyone that the manual will be discontinued. You can still buy it and the download links will stay valid until the end of this year. Thank you to everyone who bought the book and especially to those who sent me feedback and typo fixes.
To end on a positive note, there has been a final price reduction and the manual can now be bought for just €19.95.
FAQ: Will the manual be available under a free license after sales end?
Tuesday, October 20, 2020
Cargo-style dependency management for C, C++ and other languages with Meson
My previous blog post about modern C++ got a surprising amount of feedback. Some people even reimplemented the program in other languages, including one in Go, two different ones in Rust and even this slightly brain bending C++ reimplementation as a declarative style pipeline. It also got talked about on Reddit and Hacker news. Two major comments that kept popping up were the following.
- There are potential bugs if the program is ever extended to support non-ASCII text
- It is hard to use external dependencies in C++ (and by extension C)
icu_dep = dependency('icu-i18n')
thread_dep = dependency('threads')
executable('cppunicode', 'cppunicode.cpp',
dependencies: [icu_dep, thread_dep])
The threads dependency is for the multithreaded parts (see end of this post). I developed this on Linux and used the convenient system provided ICU. Windows and Mac do not provide system libs so we need to build ICU from scratch on those platforms. This is achieved by running the following command in your project's source root:
Installed icu branch 67.1 revision 1
This contacts Meson's WrapDB server and downloads build definition files for ICU. That is all you need to do. The build files do not need any changes. When you start building the project, Meson will automatically download and build the dependency. Here is a screenshot of the download step:
Here it is building in Visual Studio:
And here is the final built result running on macOS:
One notable downside of this approach is that WrapDB does not have all that many packages yet. However I have been told that given the next Meson release (in a few weeks) and some upstream patches, it is possible to build the entire GTK widget toolkit as a subproject, even on Windows.
If anyone wants to contribute to the project, contributions are most welcome. You can for example convert existing projects and submit them to wrapdb or become a reviewer. The Meson web site has the relevant documentation.
Appendix: making it parallel
Thursday, October 15, 2020
Does C++ still deserve the bad rap it has had for so long?
Traditionally C++ has been seen by many (and you know who you are) as just plain bad: the code is unreadably verbose, error messages are undecipherable, it's unsafe, compilation takes forever and so on. In fact making fun of C++ is even a fun pastime for some. All of this was certainly true in the 90s and even as recent as 10 years ago. But is it still the case? What would be a good way to determine this?
A practical experiment
Find all files with the extension .txt recursively in the subdirectories of the current directory, count the number of times each word appears in them and print the ten most common words and the number of times they are used.
We're not going to go for extreme performance or handling all possible corner cases, instead going for a reasonable implementation. The full code can be found by following this link.
The first thing we need is a way to detect files with a given extension and words within text. This calls for regular expressions:
const std::regex fname_regex("\\.txt$", std::regex_constants::icase);
const std::regex word_regex("([a-z]{2,})", std::regex_constants::icase);







