perjantai 9. elokuuta 2019

C++ modules with a batch compiler feasibility study

In our previous blog post an approach for building C++ modules using a batch compiler model was proposed. The post raised a lot of discussion on Reddit. Most of it was of the quality one would expect, but some interesting questions were raised.

How much work for compiler developers?

A fairly raised issue was that the batch approach means adding new functionality to the compiler. This is true, but not particulary interesting as a standalone statement. The real question is how much more work is it, especially compared to the work needed to support other module implementation methods.

In the batch mode there are two main pieces of work. The first is starting compiler tasks, detecting when they freeze due to missing modules and resuming them once the modules they require have been built. The second one is calculating the minimal set of files to recompile when doing incremental builds.

The former is the trickier one because no-one has implemented it yet. The latter is a well known subject, build systems like Make and Ninja have done it for 40+ years. To test the former I wrote a simple feasibility study in Python. What it does is generate 100 source files containing modules that call each other and then compiles them all in the manner a batch compiler would. There is no need to scan the contents of files, the system will automatically detect the correct build order or error out if it can not be done.

Note that this is just a feasibility study experiment. There are a lot of caveats. Please go through the readme before commenting. The issue you want to raise may already be addressed there. Especially note that it only works with Visual Studio.

The code that is responsible for running the compile is roughly 60 lines of Python. It is conceptually very simple. A real implementation would need to deal with threading, locking and possibly IPC, which would take a fair bit of work.

The script does not support incremental builds. I'd estimate that getting a minimal version going would take something like 100-200 lines of Python.

I don't have any estimates on how much work this would mean on a real compiler code base.

The difference to scanning

A point raised in the Reddit discussion is that there is an alternative approach that uses richer communication between the build system and the compiler. If you go deeper you may find that the approaches are not that different after all. It's more of a question of how the pie is sliced. The scanning approach looks roughly like this:


In this case the build system and compiler need to talk to each other, somehow, via some sort of a protocol. This protocol must be implemented by all compilers and all build tools and they must all interoperate. It must also remain binary stable and all that. There is a proposal for this protocol. The specification is already fairly long and complicated especially considering that it supports versioning so future versions may be different in arbitrary ways. This has an ongoing maintenance cost of unknown size. This also complicates distributed builds because it is common for the build system and the compiler to reside on different machines or even data centres so setting up an IPC channel between them may take a fair bit of work.

The architectural diagram of the batch compiler model looks like this:


Here the pathway for communicating module information is a compiler implementation detail. Every compiler can choose to implement it in the way that fits their internal compiler implementation the best. They can change its implementation whenever they wish because it is never exposed outside the compiler. The only public facing API that needs to be kept stable is the compiler command line, which all compilers already provide. This also permits them to ship a module implementation faster, since there is no need to fully specify the protocol and do interoperability tests. The downside is that the compiler needs to determine which sources to recompile when doing incremental builds.

Who will eventually make the decision?

I don't actually know. But probably it will come down to what the toolchain providers (GCC, Clang, Visual Studio) are willing to commit to.

It is my estimate (which is purely a guess, because I don't have first hand knowledge) that the batch system would take less effort to implement and would present a smaller ongoing maintenance burden. But compiler developers are the people who would actually know.

keskiviikko 7. elokuuta 2019

Building C++ modules, take N+1

Modules were voted in C++20 some time ago. They are meant to be a replacement for #include statements to increase build speeds and to also isolate translation units so, for example, macros defined in one file do not affect the contents of another file. There are three major different compilers and each of them has their own prototype implementation available (GCC documentation, Clang documentation, VS documentation).

As you would expect, all of these implementations are wildly different and, in the grand C++ tradition, byzantinely complicated. None of them also have a really good solution to the biggest problem of C++ modules, namely that of dependency tracking. A slightly simplified but mostly accurate description of the problem goes like this:

Instead of header files, all source code is written in one file. It contains export statements that describe what functions can be called from the outside. An analogy would be that functions declared as exported would be in a public header file and everything else would be internal and declared in an internal header file (or would be declared static or similar). The module source can not be included directly, instead when you compile the source code the compiler will output an object file and also a module interface file. The latter is just some sort of a binary data file describing the module's interface. An import statement works by finding this file and reading it in.

If you have file A that defines a module and file B that uses it, you need to first fully compile file A and only after the module interface file has been created can you compile file B. Traditionally C and C++ files can be compiled in parallel because everything needed to compile each file is already in the header files. With modules this is no longer the case. If you have ever compiled Fortran and this seems familiar, it's because it is basically the exact same architecture.

Herein lies the problem

The big, big problem is how do you determine what order you should build the sources in. Just looking at the files is not enough, you seemingly need to know something about their contents. At least the following approaches have toyed with:
  1. Writing the dependencies between files manually in Makefiles. Yes. Really. This has actually been but forth as a serious proposal.
  2. First scan the contents of every file, determine the interdependencies, write them out to a separate dependency file and then run the actual build based on that. This requires parsing the source files twice and it has to be done by the compiler rather than a regex because you can define modules via macros (at least in VS currently).
  3. When the compiler finds a module import it can not resolve, it asks the build system via IPC to generate one. Somehow.
  4. Build an IPC mechanism between the different compiler instances so they can talk to each other to determine where the modules are. This should also work between compilers that are in different data centers when doing distributed builds.
Some of these approaches are better than others but all of them fail completely when source code generators enter the picture, especially if you want to build the generator executable during the build (which is fairly common). Scanning all file contents at the start of the build is not possible in this case, because some of the source code does not yet exist. It only comes into existence as build steps are executed. This is hideously complicated to support in a build system.

Is there a better solution?

There may well be, though I'd like to emphasize that none of the following has actually been tested and that I'm not a compiler developer. The approach itself does require some non-trivial amount of work on the compiler, but it should be less than writing a full blown IPC mechanism and distributed dataflow among the different parts of the system.

At the core of the proposed approach is the realization that not all module dependencies between files are the same. They can be split into two different types. This is demonstrated in the following diagram that has two targets: a library and an executable that uses it.


As you can see the dependencies within each target can get fairly complicated. The dependencies between targets can be just as complicated, but they have been left out of the picture to keep it simple. Note that there are no dependency cycles anywhere in the graph (this is mandated by the module specification FWICT). This gives us two different kinds of module dependencies: between-targets module dependencies and within-targets module dependencies.

The first one of these is actually fairly simple to solve. If you complete all compilations (but not the linking step) of the dependency library before starting any compilations in the executable, then all library module files that the executable could possibly need are guaranteed to exist. This is easy to implement with e.g. a Ninja pseudotarget.

The second case is the difficult one and leads to all the scanning problems and such discussed above. The proposed solution is to slightly change the way the compiler is invoked. Rather than starting one process per input file, we do something like the following:

g++ <other args> --outdir=somedir [all source files of this target]

What this means conceptually is that the compiler needs to take all the input files and compile each of them. Thus file1.cpp should end up as somedir/file1.o and so on. In addition it must deal with this target's internal module interrelations transparently behind the scenes. When run again it must detect which output files are up to date and rebuild only the outdated ones.

One possible implementation is that the compiler may launch one thread per input file (but no more than there are CPUs available). Each compilation proceeds as usual but when it encounters a module import that it can not find, it halts and waits on, say, a condition variable. Whenever a compilation job finishes writing a module, it will signal all the other tasks that a new module is available. Eventually either all jobs finish or every remaining task is deadlocked because they need a module that can't be found anywhere.

This approach is similar to the IPC mechanism described on GCC's documentation but it is never exposed to any third party program. It is fully an internal implementation detail of the compiler and as such there are no security risks or stability requirements for the protocol.

With this approach we can handle both internal and external module dependencies reliably. There is no need to scan the sources twice or write complicated protocols between the compiler and the build system. This even works for generated sources without any extra work, which no other proposed approach seems to be able to do.

As an added bonus the resulting command line API is so simple it can be even be driven with plain Make.

Extra bits

This approach also permits one to do ZapCC style caching. Since compiler arguments for all sources within one target must be the same under this scheme (which is a good thing to do in general), imports and includes can be potentially shared between different compiler tasks. Even further, suppose you have a type that is used in most sources like std::vector<std::string>.  Normally the instantiated and compiled code would need to be written in every object file for the linker to eventually deduplicate. In this case, since we know that all outputs will go to the same target it is enough to write the code out in one object file. This can lead to major build footprint reductions. It should also reduce the linker's memory usage since there is a lot less debug info to manage. In most large projects linking, rather than compiling, is the slow and resource intensive step so making it faster is beneficial.

The module discussions have been going around in circles about either "not wanting to make the compiler a build system" or "not wanting to make the build system into a compiler". This approach does neither. The compiler is still a compiler, it just works with slightly bigger work chunks at a time. It does need to track staleness of output objects, though, which it did not need to do before.

There needs to be some sort of a load balancing system so that you don't accidentally spawn N compile jobs each of which spawns N internal work threads.

If you have a project that consists of only one executable with a gazillion files and you want to use a distributed build server then this approach is not the greatest. The solution to this is obvious: split your project into independent logical chunks. It's good design and you should do it regardless.

The biggest downside of this approach that I could come up with was that CCache will probably no longer work without a lot of work. But if modules make compilation 5-10× faster (which is a given estimate, there are no public independent measurements yet) then it could be worth it.

torstai 1. elokuuta 2019

Metafont-inspired font design using nonlinear constraint optimization and Webassembly

Modern fonts work by drawing the outline of each letter out by hand. This is a very labour-intensive operation as each letter has to be redrawn for each weight. Back in the late 70s Donald Knuth created METAFONT, which has a completely different approach that mimics the way letters are drawn by hand. In this system you specify a pen shape and a stroke path. The computer would then calculate what sort of a mark this stroke would leave on paper and draw the result. The pen shape as well as the stroke were defined as mathematical equations of type "the stroke shall begin so that its topmost part is exactly at x-height and shall end so that its bottom is at the baseline".

The advantage of this approach is that it is fully parametrizable. If you change the global x-height, then every letter is automatically recalculated. Outline fonts can't be slanted by shearing because it changes the strokes' widths. Parametric fonts do not have this issue. Different optical weights can be obtained simply by changing the size of the pen nib. You could even go as far as change the widths of individual letters during typesetting to make text justification appear even smoother.

METAFONT was never widely used for font design. The idea behind it was elegant, though, and is worth further examination, for example are there other ways of defining fonts parametrically and what sort of letter shapes would they produce. Below we describe one such attempt.

How would that look in practice?

In a nutshell you write a definition that mathematically specifies the shape:


Then the system will automatically create the "best possible" shape that fulfills those requirements:


Note that this output comes from the Python Scipy prototype version, not the Webassembly one. It calculates a slightly different shape for the letter and does not do the outline yet.

How does it work?

First the program parses the input file, which currently can specify only one stroke. Said stroke consists of one or more cubic splines one after the other. That is, the end point of spline n is the starting point of spline n+1. It then converts the constraints into mathematical form. For example if you have a constraint that a point must be on a diagonal line between (0, 0) and (1, 1), then its coordinates must be (t, t) where t is a free parameter with values in the range [0, 1]. The free parameters can be assigned any values and the resulting curve will still fulfill all the requirements.

The followup question is how do we select the free parameters to pick the most pleasing of all curves to quote Knuth's terminology. If you thought "deep learning" then you thought wrong. This is actually a well known problem called nonlinear optimization. There are many possible approaches but we have chosen to use the (draws a deep breath) Broyden–Fletcher–Goldfarb–Shanno algorithm. This is the default optimization method in Scipy. There is also a C port of the original Fortran implementation.

The last thing we need is a target function to minimize. There are again many choices (curve length, energy, etc). After some experimentation we chose the maximum absolute value of the second derivative's perpendicular component over the entire curve. It gave the smoothest end result. Having defined a target function, free variables and a solver method we can let the computer do what it does best: crunch numbers. The end result is written as SVG using TinyXML2.

There are known problems with the current evaluation algorithm. Sometimes it produces weird and non-smooth results. LBFGS seems to be sensitive to the initial estimate and there are interesting numerical problems when evaluating the target function when switching between two consecutive curves.

Try it yourself

The Webassembly version is available here. You can edit the definition text and then recompute the letter shape with the action button. Things to try include changing the width and height (variables w and h, respectively) or changing the location constraints for curve points.

The code can also be compiled and run as a native program. It is available in this Github repository.

Future work and things that need fixing

  • Firefox renders the resulting SVG files incorrectly
  • All mathy bits are hardcoded, optimization functions et al should be definable in the character file instead
  • Designing strokes by hand is not nice, some sort of a UI would be useful
  • The text format for specifying strokes needs to be designed
  • Should be able to directly specify some path as a straight line segment rather than a bezier
  • You should be able to specify more than one stroke in a character
  • Actual font file generation, possibly via FontForge
  • "Anti-drawing" or erasing existing strokes with a second stroke

tiistai 16. heinäkuuta 2019

A personal story about 10× development

During the last few days there has been an ongoing Twitter storm about 10× developers. And like all the ones before it (and all the future ones that will inevitably happen) the debate immediately devolved into name calling and all the other things you'd except from Twitter fights. This blog post is not about that. Instead it is about a personal experience about productivity that I had to experience closer than I would have liked.

Some years ago I was working for company X on product Y. All in all it was quite a nice experience. We had a small team working on a code base that was pretty good. It had nice tests, not too many bugs, and when issues did arise they were usually easy to fix. Eventually the project was deemed good enough and we were transferred to work on different projects.

I have no idea what our "industry standard performance multiplier" was when we worked on that project, but for the sake of argument let's call it 1×.

The project I got transferred to was the thing of nightmares. It was a C++ project and all the bad things that have ever been said about C++ were true about that code base. There was not much code but it was utterly incomprehensible. There were massively deep inheritance hierarchies, , compilation speed was measured in minutes for even the most trivial changes, and so on. It was managed by an architecture astronaut that, as one is wont to do, rewrote existing mature libraries as header only template libraries that were buggy and untested (one could even say untestable).

Thus overnight I went from being a 1× down to being a 0.1× or possibly even a 0.01× developer. Simply trying to understand what a specific function was supposed to do took hours. There was, naturally, a big product launch coming up so we needed to get things finished quickly. All in all it was a stressful, frustrating and unpleasant situation to be in. And that was not the worst of it.

After a few weeks my manager wanted to talk to me in private. He was very concerned about the fact that I had not achieved any visible progress for a while. Then I explained to him in detail all the problems in the current project. I even demonstrated how compiling a simple helloworld-level program with the frameworks we had to use took tens of seconds on the beefiest i7 desktop machine I had available. He did not seem to be able to grasp any of that as his only response was "but you used to be so productive in your previous project". Shortly thereafter the same boss started giving me not-al-all-thinly-veiled accusations that I was just slacking off and that this could lead to serious reprimands.

This story does not have a happy ending. The project eventually failed (due to completely different reasons, though), money was squandered and almost everyone involved got fired. In the aftermath I seriously considered getting out of the software engineering business altogether. The entire experience had been so miserable that becoming a 0× developer was seriously tempting.

Is there something we can learn from this?

The "×ness" of any developer does not exist in a vacuum but depends on many organizational things. The most obvious one is tooling. If you have a CI where tests take 30 minutes to run or your developers have underpowered old laptops, everyone's performance goes down. In fact, the overall health of the code base probably has a bigger effect on developer productivity than all developers' skills combined.

But even more important than technical issues are things that promote healthy team dynamics. These include things like blameless postmortems, openness to ideas from everyone, permission to try new things even if they may fail, stern weeding out of jerk behaviour and, ultimately, trust.

If you work on getting all of these things in your working environment then you may find that you find yourself with a 10× team. And if you do, the entire concept of a single 10× developer becomes meaningless.

sunnuntai 14. heinäkuuta 2019

Initializing all local variables with Clang-Tidy

A common source of all kinds of bugs is using variables without properly initializing them. Out of all security problems this one is the simplest to fix, just convert all declarations of type int x; to int x=0;. The main reason for not doing that is laziness, manually going through existing code bases and adding initialization statements is boring and nobody wants to do that.

Fortunately nowadays we don't have to. Clang-tidy provides a nice toolkit for writing source code refactoring tools for C and C++. As an exercise I wrote a checker to do this. It is submitted upstream and is undergoing code review. Implementing it was fairly straightforward. There were only two major problems. The first one was that existing documentation consists mostly of reference manuals. There is no easy to follow tutorials, only Doxygen pages. But if you dig around on the net and work on it a bit, you can get it working.

The second, and bigger, obstacle is that doing anything in the LLVM code base is sloooow. Everything in LLVM and Clang is linked to single, huge, monolithic libraries which take forever to link. Because of reasons I started doing this work on my secondary machine, which is a 4 core i5 with 16 gigs of RAM. I had to limit simultaneous linker jobs to 2 because otherwise it would just crash spectacularly to an out of memory error. Presumably it is impossible to compile the code base on a machine that has only 8 gigs of RAM. It seems that if you want to do any real development on LLVM you need a spare data center to run the compilations, which is unfortunate.

There is an evil hack to work around this, though. Set the CMake build type to Debug and then change CMAKE_CXX_FLAGS_DEBUG and CMAKE_C_FLAGS_DEBUG from -g to -Og. This makes the compilation faster and reduces memory usage to a fraction of the original. The downside is that there is no debug information, but it turns out to not be necessary when writing simple Clang-Tidy checkers.

Once all that is done the actual checker is almost trivial. This is the part that looks up all local variables without initial values:

void InitLocalVariablesCheck::registerMatchers(MatchFinder *Finder) {
  Finder->addMatcher(
      varDecl(unless(hasInitializer(anything()))).bind("vardecl"), this);
}

Then you determine what the initial value should be based on the type of the variable and add a warning and a fixit:

diag(location, "variable %0 is not initialized")
    << MatchedDecl;
diag(location, "insert initial value", DiagnosticIDs::Note)
    << FixItHint::CreateInsertion(
           location.getLocWithOffset(VarName.size()),
           InitializationString);

All in all this amounts to about 100 lines of code plus tests.

But what about performance?

The other major reason not to initialize variables is that it "may cause a runtime performance degradation of unknown magnitude". That is true, but with this tooling the degradation is no longer unknown. You can run the tool and then measure the results. This is trivial for all code bases that have performance benchmarks.

perjantai 7. kesäkuuta 2019

Tweaking the parallel Zip file writer

A few years ago I wrote a command line program to compress and decompress Zip files in parallel. It turned out to work pretty well, but it had one design flaw that kept annoying me.

What is the problem?

Decompressing Zip files in parallel is almost trivial. Each file can be decompressed in parallel without affecting any other decompression task. Fire up N processing tasks and decompress files until finished. Compressing Zip files is more difficult to parallelize. Each file can be compressed separately, but the problem comes from writing the output file.

The output file must be written one file at a time. So if one compressed file is being written to the output file then other compression tasks must wait until it finishes before their output can be written to the result file. This data can not be kept in memory because it is common to have output files that are larger than available memory.

The original solution (and thus the design flaw alluded to) was to to have each compressor write its output to a temporary file. The writer would then read the data from the file, write it to the final result file and delete the temporary file.

This works but means that the data gets written to the file system twice. It may also require up to 2× disk space. The worst case happens when you compress only one very big file. On desktop machines this is not such a problem, but on something like a Raspberry Pi the disk is an SD card, which is very slow. You only want to write it once. SD cards also wear out when written to, which is another reason to avoid writes.

The new approach

An optimal solution would have all of these properties:
  1. Uses all CPU cores 100% of the time (except at the end when there are fewer tasks than cores).
  2. Writes data to the file system only once.
  3. Handles files of arbitrary size (much bigger than available RAM).
  4. Has bounded memory consumption.
It turns out that not all of these are achievable at the same time. Or at least I could not come up with a way. After watching some Numberphile videos I felt like writing a formal proof but quickly gave up on the idea. Roughly speaking since you can't reliably estimate when the tasks finish and how large the resulting files will be, it does not seem possible to choose an optimal strategy for writing the results out to disk.

The new architecture I came up with looks like this:


Rather than writing its result to a temporary file, each compressor writes it to a byte queue with a fixed maximum size. This was chosen to be either 10 or 100 megabytes, which means that in practice most files will fit the buffer. The queue can be in one of three states: not full, full or finished. The difference between the last two is that a full queue is one where the compression task still has data to compress but it can't proceed until the queue is emptied.

The behaviour is now straightforward. First launch compressor tasks as in decompressing. The file writer part will go through all the queues. If it finds a finished queue it will write it to disk and launch a new task. If it finds a full queue it will do the same, but it must write out the whole stream, meaning it is blocked until the current file has been fully compressed. If the compressions takes too long all other compression tasks will finish (or get full) but new ones can't be launched leading to CPU underutilization.

Is it possible to do better?

Yes, but only as a special case. Btrfs supports copying data from one file to another in O(1) time taking only an extra O(1) space. Thus you could write all data to temp files, copy the data to the final file and delete the temp files.

lauantai 1. kesäkuuta 2019

Looking at why the Meson crowdfunding campaign failed

The crowdfunding campaign to create a full manual for the Meson build system ended yesterday. It did not reach its 10 000€ goal so the book will not be produced and instead all contributed money will be returned. I'd like to thank everyone who participated. A special thanks goes out to Centricular for their bronze corporate sponsorship (which, interestingly, was almost 50% of the total money raised).

Nevertheless the fact remains that this project was a failure and a fairly major one at that since it did not reach even one third of its target. This can not be helped, but maybe we can salvage some pieces of useful information from the ruins.

Some statistics

There were a total of 42 contributors to the campaign. Indiegogo says that a total of 596 people visited the project when it was live. Thus roughly 7% of all people who came to the site participated. It is harder to know how many people saw information about the campaign without coming to the site. Estimating based on numbers based on the blog's readership, Twitter reach and other sources puts the number at around 5000 globally (with a fairly large margin of error). This would indicate a conversion rate of 1% of all the people who saw any information about the campaign. In reality the percentage is lower since many of the contributors were people who did not really need convincing. Thus the conversion rate is probably closer to 0.5% or even lower.

The project was set up so that 300 contributors would have been enough to make the project a success. Given the number of people using Meson (estimated to be in the tens of thousands) this seemed like a reasonable goal. Turns out that it wasn't. Given these conversion numbers you'd need to reach 30 000 – 60 000 people in order to succeed. For a small project with zero advertising budget this seems like a hard thing to achieve.

On the Internet everything drowns

Twitter, LinkedIn, Facebook and the like are not very good channels for spreading information. They are firehoses where any one post has an active time of maybe one second if you are lucky. And if you are not, the platforms' algorithms will hide your post because they deem it "uninteresting".  Sadly filtering seems to be mandatory, because not having it makes the firehose even more unreadable. The only hope you have is that someone popular writes about your project. In practice this can only be achieved via personal connections.

Reddit-like aggregation sites are not much better, because you have basically two choices: either post on a popular subreddit or an unpopulare one. In the first case your post probably won't even make it on the front page, all it takes is a few downvotes because the post is "not interesting" or "does not belong here". A post that is not on the front page might not as well even exist; no-one will read it. Posting on an non-popular area is no better. Your post is there but it will reach 10 people and out of those maybe 1 will click on the link.

New sites are great for getting the information out, but they suffer from the same popularity problem as everything else. A distilled (and only slightly snarky) explanation is that news sites write mainly about two things:
  1. Things they have already written about (i.e. have deemed popular)
  2. Things other news sites write about (i.e. that other people have deemed popular)
This is not really the fault of news sites. They are doing their best on a very difficult job. This is just how the world and popularity work. Things that are popular get more popular because of their current popularity alone. Things that are not popular are unlikely to ever become popular because of their current unpopularity alone.

Unexpected requirements

One of the goals of this campaign (or experiment, really) was to see if selling manuals would be a sustainable way to compensate FOSS project developers and maintainers for their work. If working this would be a good way for compensation, because there are already established legal practices for selling books across the world. Transferring money in other ways (donations etc) is difficult and there may be legal obstacles.

Based on this one experiment this does not seem to be a feasible approach. Interestingly multiple people let me know that they would not be participating because the end result would not be released under a free license. Presumably the same people do not complain to book store tellers that "I will only buy this Harry Potter book if, immediately after my purchase, the full book is released for free on the Internet". But for some reason there is a hidden assumption that because a person has released something under a free license, they must publish everything else they do under free licenses as well.

These additional limitations make this model of charging for docs really hard to pull off. There is no possibility of steady, long term money flow because once a book is out under a free license it becomes unsellable. People will just download the free PDF instead. A completely different model (or implementation of the attempted model) seems to be needed.

So what happens next?

I don't really know. Maybe the book can get published through an actual publisher. Maybe not. Maybe I'll just take a break from the whole thing and get back to it later. But to end on some kind of a positive note I have extracted one chapter from the book and have posted it here in PDF form for everyone to read. Enjoy.

sunnuntai 12. toukokuuta 2019

Emulating rpath on Windows via binary patching

A nice feature provided by almost every Unix system is rpath. Put simply it is a way to tell the dynamic linker to look up shared libraries in custom directories. Build systems use it to be able to run programs directly from the build directory without needing to fiddle with file copying or environment variables. As is often the case, Windows does things completely differently and does not have a concept of rpath.

In Windows shared libraries are always looked up in directories that are in the current PATH. The only way to make the dynamic linker look up shared libraries in other directories is to add them to the PATH before running the program. There is also a way to create a manifest file that tells the loader to look up libraries in a special place but it is always a specially named subdirectory in the same directory as the executable. You can't specify an arbitrary path in the manifest, so the libraries need to be copied there. This makes Windows development even more inconvenient because you need to either fiddle with paths, copy shared libraries around or statically link everything (which is slooooow).

If you look at Windows executables with a hex editor, you find that they behave much the same way as their unixy counterparts. Each executable contains a list of dependency libraries that it needs, such as helper.dll. Presumably what happens is that at runtime the dynamic linker will parse the exe file and pass the library names to some lookup function that finds the actual libraries given the current PATH value. This raises the obvious question: what would happen if, somehow, the executable file would have an absolute path written in it rather than just the filename?

It turns out that it works and does what you would expect it to. The backend code accepts absolute paths and resolves them to the correct file without PATH lookups. With this we have a working rpath simulacrum. It's not really workable, though, since the VS toolchain does not support writing absolute paths to dependencies in output files. Editing the result files by hand is also a bit suspicious because there are many things that depend on offsets inside the file. Adding or removing even one byte will probably break something. The only thing we can really do is to replace one string with a different one with the same length.

This turns out to be the same problem that rpath entries have on Unix and the solution is also the same. We need to get a long enough string inside the output file and then we can replace it with a different string. If the replacement string is shorter, it can be padded with null bytes because the strings are treated as C strings. I have written a simple test repository doing this, which can be downloaded from Github.

On unix rpath is specified with a command line argument so it can be padded to arbitrary size. Windows does not support this so we need to fake it. The basic idea is simple. Instead of creating a library helper.dll we create a temporary library called aaaaaaaaaaaaaaaaaaaaaaaa.dll and link the program against that. When viewed in a hex editor the executable looks like this.


Now we can copy the library to its real name in a subdirectory and patch the executable. The result looks like this.


The final name was shorter than what we reserved so there are a bunch of zero bytes in the executable. This program can now be run and it will always resolve to the library that we specified. When the program is installed the entry can be changed to just plain helper.dll in the same way making it indistinguishable from libraries built without this trick (apart from the few extra null bytes).

Rpath on Windows: achieved.

Is this practical?

It's hard to say. I have not tested this on anything except toy programs but it does seem to work. It's unclear if this was the intended behaviour, but Microsoft does take backwards compatibility fairly seriously so one would expect it to keep working. The bigger problem is that the VS toolchain creates many other files, such as pdb debug info files, that probably don't like being renamed like this. These files are mostly undocumented so it's difficult to estimate how much work it would take to make binary hotpatching work reliably.

The best solution would be for Microsoft to add a new linker argument to their toolchain that would write dependency info to the files as absolute paths and to provide a program to rewrite those entries as discussed above. Apple already provides all of this functionality in their core toolchain. It would be nice for MS to do the same. This would simplify cross platform development because it would make all the major platforms behave in the same way.

keskiviikko 8. toukokuuta 2019

Why crowdfunding freely licensed documentation is illegal in Finland

On the Meson manual crowdfunding page it is mentioned that the end result can not be put under a fully free license. Several people have said that they "don't believe such a law could exist" or words to that effect. This blog post is an attempt to to explain the issue in English as all available text about the case is in Finnish. As a disclaimer: I'm not a lawyer, the following is not legal advice, there is no guarantee, even that any of the information below is factual.

To get started we need to go back in time a fair bit and look at disaster relief funds. In Finland you must obtain a permit from the police in order to gather money for general charitable causes. This permit has strict requirements. The idea is that you can't just start a fundraising, take people's money and pocket it, instead the money must provably go to the cause it was raised for. The way the law is written is that a donation to charity is done without getting "something tangible" in return. Roughly if you give someone money and get a physical item in return, it is considered a sales transaction. If you give money to someone and in return get a general feeling of making the world better in some way, that is considered a donation. The former is governed by laws of commerce, the latter by laws of charity fundraising.

A few years ago there was a project to create a book to teach people Swedish. The project is page is here, but it is all in Finnish so it's probably not useful to most readers. They had a crowdfunding project to finish the project with all the usual perks. One of the goals of the crowdfunding was to make the book freely distributable after publishing. This is not unlike funding feature work on FOSS projects works.

What happened next is that the police stepped in and declared this illegal (news story, in Finnish). Their interpretation was that participating in this campaign without getting something tangible in return (i.e. paying less than the amount needed to get the book) was a "charitable donation". Thus it needs a charity permit as explained above. Running a crowdfunding campaign is still legal if it is strictly about pre-sales. That is, every person buys "something" and that something needs to have "independent value" of some sort. If the outcome of a project is a PDF and that PDF becomes freely available, it can be argued that people who participated did not get any "tangible value" in exchange for their money.

Because of this the outcome of the Meson manual crowdfunding campaign can not be made freely available. This may seem a bit stupid, but sadly that's the law. The law is undergoing changes (see here, in Finnish), but those changes will not take effect for quite some time and even when they do it is unclear how those changes would affect these kinds of projects.

keskiviikko 1. toukokuuta 2019

The Meson manual crowdfunding campaign

The Meson Build system has existed for over six years. Even though we have a fairly good set of documentation, there has not been a standalone user's manual for it. Until now.


A crowdfunding campaign to finance the manual has just been launched on Indiegogo. The basic deal is simple, for 30€ you get the final book as a PDF. To minimize work and save trees, there is no physical version. There are also no stickers, beer mats or any other tchotchkies. There are a few purchase options as well as opportunities for corporate sponsorships. Please see the Indiegogo project page for further details. If there are any questions about this campaign feel free to contact me. The easiest way is via email.

Overall I'm quite excited about this campaign. One reason is obviously personal, but the other has to do with sustainability of FOSS projects in general. There has been a lot of talk about how maintainers of open source projects can get compensated for their work. This campaign can be seen as an experiment to see if the crowdfunding model could work in practice.

So if you are just getting started with building software and want a user manual, buy this book. If you have basic experience with Meson and want to dive deeper, buy this book. If you are a seasoned veteran and don't really need a book but want to support the project (specifically me), buy this book. Regardless of anything else, please spread the word on your favourite social media and real world venues of choice.

Let the experiment begin!

maanantai 15. huhtikuuta 2019

An important message to people designing testing frameworks!

Do not, I repeat, NOT make your test framework fail a test run if it writes any text to stderr! No matter how good of on idea you think it is, it's terrible.

If you absolutely, positively have to do that, then print the reason for this failure in your output log. If you can't think of a proper warning message, feel free to copy paste this one:

THIS TEST FAILED BECAUSE IT WROTE TO STDERR AND SOMEONE HERE (OBVIOUSLY NOT ME) THOUGHT MAKING THAT A HARD ERROR WOULD BE A GOOD IDEA!!!!!!!

Sincerely: a person who has lost hours of his life on this sh*t on multiple occasions and can never get it back.

tiistai 26. helmikuuta 2019

Could Linux be made "the best development environment" for games?

It is fairly well established that Linux is not the #1 game development platform at this point in time. It is also unlikely to be the most popular one any time soon due to reasons of economics and inertia. A more interesting question, then, would be can it be made the best one? The one developers favour over all others? The one that is so smooth and where things work so well together that developers feel comfortable and productive in it? So productive that whenever they use some other environment, they get aggravated by being forced to use such a substandard platform?

Maybe.

The requirements

In this context we use "game development" as a catch all term for software development that has the following properties:
  1. The code base is huge, millions or tens of millions lines of code.
  2. Non-code assets are up to tens of gigabytes in size
  3. A program, once built, needs to be tested on a bunch of various devices (for games, using different 3D graphics cards, amounts of memory, processors etc).

What we already have

A combination of regular Linux userland and Flatpak already provides a lot. Perhaps the most unique feature is that you can get the full source code of everything in the system all the way down to the graphic cards' device drivers (certain proprietary hardware vendors notwithstanding). There is no need to guess what is happening inside the graphics stack, you can just add breakpoints and step inside it with full debug info.

Linux as a platform is also faster than competing game development systems at most things. Process invocation is faster, file systems are faster, compilations are faster, installing dependencies is faster. These are the sorts of small gains that translate to better productivity and developer happiness.

Flatpak as a deployment mechanism is also really nice.

What needs to be improved?

Many tools in the Linux development chain assume (directly or indirectly) that doing something from scratch for every change is "good enough". Once you get into large enough scale this no longer works. As an example flatpak-builder builds its packages by copying the entire source tree inside the build container. If your repository is in the gigabyte range this does not work, but instead something like bind mounting should be used. (AFAIK GNOME Builder does something like this already.) Basically every operation needs to be O(delta) instead of O(whole_shebang):
  • Any built program that runs on the developer's machine must be immediately deployable on test machines without needing to do a rebuild on a centralised build server.
  • Code rebuilds must be minimal.
  • Installs must skip files that have not changed since the last install.
  • Package creation must only account for changed files.
  • All file transfers must be delta based. Flatpak already does this for package downloads but building the repo seems to take a while.
A simple rule of thumb for this is that changing one texture in a game and deploying the result on a remote machine should not take more than 2x the amount of time it would take to transfer the file directly over with scp.

Other tooling support

Obviously there needs to be native support for distributed builds. Either distcc, IceCream or something fancier, but even more important is great debugging support.

By default the system should store full debug info and corresponding source code. It should also log all core dumps. Pressing one button should then open up the core file in an IDE with up to date source code available and ready to debug. This same functionality should also be usable for debugging crashes in the field. No crash should go unstored (assuming that there are no privacy issues at play).

Perhaps the hardest part is the tooling for non-coders. It should be possible to create new game bundles with new assets without needing to touch any "dev" tools, even when running a proprietary OS. For example there could be a web service where you could do things like "create new game install on machine X and change model file Y with this uploaded file Z". Preferably this should be doable directly from the graphics application via some sort of a plugin. 

Does something like this already exist?

Other platforms have some of these things built in and some can be added with third party products. There are probably various implementations of these ideas inside the closed doors of many current game development studios. AFAICT there does not exist a fully open product that would combine all of these in a single integrated whole. Creating that would take a fair bit of work, but once done we could say that the simplest way to set up the infrastructure to run a game studio is to get a bunch of hardware, open a terminal and type:

sudo apt install gamestudio

lauantai 23. helmikuuta 2019

What if everything you know is wrong?

In interesting intellectual design challenge is to take a working thing (library, architecture, etc) and then see what would happen if you would reimplement it with the exact opposite way. Not because you'd use the end result anywhere, but just to see if you can learn something new. Or, in other words:
Screenshot of Marble Madness Silly Race level with text "Everything you know is wrong".
As an example let's apply this approach to C++'s fixed size array object or std::array. Some of its core design principles include:
  1. Indexing is by default unsafe, user is responsible for providing valid indexes.
  2. Errors are reported via exceptions.
  3. Iterators may be invalid and invoking invalid iterators is allowed but UB.
  4. Indexing must be exactly as fast as accessing a C array.
Inverting all of these give us the following design principles:
  1. Out of bound accesses must be impossible.
  2. No exceptions (assuming contained objects do not throw exceptions).
  3. All iterator dereferences are guarded and may never lead to bad accesses.
  4. It's ok to use some (but not much) processor time to ensure safety. Aim for zero overhead when possible.

So how does it look like?

An experimental PoC implementation can be found in this Github repo. Note that the code is intentionally unpolished. There are silly choices made. That is totally fine, it's not meant for actual use, only to explore the problem space.

The most important operation for an array type is indexing. It must work for all index values, even for those out of bounds. As no exceptions are allowed, the natural way to make this work is to return a Maybe type. This could be a std::optional<std::reference_wrapper<T>>, but for reasons that will become apparent later, we use a custom type. The outcome for this is kind of nice, allowing you to do things like (assuming an array of size 20):

assert(x[0]);       // check if index is valid.
assert(!x[20]);     // OOB indexes are invalid
*x[0] = 5           // assignment
assert(*x[0] == 5); // dereference
int i = *x[20];     // aborts program

The overall usability is kind of nice. This is similar to languages that have int? variables. The biggest problem here is that there is no way to prevent dereferencing an invalid maybe object, leading to termination. A typical bug would look like this:

if(maybe) {
  *maybe = 3;
} else {
  *maybe = 4; // Legal code but should not be.
}

There are at least three possible architectural ways to solve this:
  1. Pattern matching (a switch/case on the object type) with language enforcement.
  2. A language construct for "if attempting to use an invalid maybe object, exit the current function with an error". There have been talks of a try statement that would do something like this.
  3. Maybes can not be dereferenced, only called with a method like visit(if_valid_function, if_not_valid_function).
#3 can be done today but is tedious and still does not permit automatically returning an error from the current function block.

Iteration

Creating a safe iterator is fairly simple. This iterator has a pointer to the original object and an integer offset. Dereferencing it calls the indexing operation and returns the maybe to the caller. This works fine until you test it with std::sort and after a lot of debugging find out that the implementation has a code block looking like this:

T tmp = std::move(a);
a = std::move(b);
b = std::move(tmp);

Even if you have swap defined for your objects, it will call this at some point. The problem here is that since the temporary does not point to any existing object, it can not store the value of the first move. There does not seem to be a good solution for this. Swap-only types might work, but such a type can not be defined with C++'s type system (or at least std::sort can not handle such types). The solution used in this example is that if a maybe object is created so that it does not point to any existing object, it stores objects inside itself. This works (for this use case) but feels a bit iffy.

Another problem this exposes is that the STL does not really work with these kinds of types. The comparison function returns a boolean, but if you compare (for whatever reason) invalid iterators and don't use exceptions there should be three different return values: true, false and erroneous_comparison. That is, an std::optional<bool>. Fixing this properly would mean changing std::sort to handle failing comparisons.

But what about performance?

For something like sorting, checking every access might cause a noticeable performance hit. If you do something like this:

std::sort(coll.begin(), coll.end())

it is fairly easy to verify that the range is valid and all thus accesses will be valid (assuming no bugs in the sort implementation). It would be nice to be able to opt out of range checks in these cases. Here's one way of doing it:

std::sort(coll.unsafe().begin(), coll.unsafe().end())

Here unsafe is a helper class that returns raw pointers to the underlying object. In this way individual accesses (which are bug prone) are guarded but larger batch operations (which are safer) can optionally request a faster code path. The unsafe request is immediately apparent and easily greppable.

Is this fully safe?

No. There are at least three different ways of getting an invalid reference.
  1. Deleting the main object while iterators/maybes are outstanding. Would require tracking all outstanding objects. Probably too heavyweight. In Rust this is handled by the borrow checker at compile time.
  2. Unallocating the memory backing the object without running its destructor. This is actually legal. Can't be guarded against.
  3. Changing the object's backing memory to read only with a syscall. Can't be guarded against.

torstai 14. helmikuuta 2019

Introducing the Meson Quest

Most FOSS projects have a bug tracker, code repository and an IRC channel. Many even have a developer meeting, conference or awards given out to most prolific contributors. But do you know what no FOSS project has that they really should?

A Dragon Ball/NES/SNES style RPG quest.

To fill this void in the FOSS experience we have taken it upon ourselves to create one. The first thing any quest needs is artifacts worth fighting over. Going with the style of Dragon Ball we have created six power glasses infused with the power of software building. This is what they look like.


These items were hand-crafted in a blazing Finnish furnace in the dead of winter. Once cast and cooled the Meson logo and the developer team's secret motto were debossed on the side by artisanal free range engraving elves of Espoo.

To make the quest more challenging, we have scattered all the power glasses to the extreme ends of the world. Each is being held by a mini boss who must be defeated in order to obtain the corresponding power. In no particular order they are:

  • Igor Gnatenko, Czech republic, holding the power of the First Community Member
  • Nirbheek Chauhan, India, holding the power of the Co-Maintainer
  • Dylan Baker, USA, holding the power of 3D Graphics
  • Alexis Jeandet, France, holding the power of CI
  • Aleksey Filippov, Ireland, holding the power of Dependency Packaging
They have all stood as guardians of their respective power for an extended amount of time and for that they deserve vast heaps of praise and gratitude from all of us.

The sixth and last power glass is held under strict supervision in my Finnish volcano lair headquarters [1] and can only be unlocked when holding all the other five power glasses.

It is currently unclear whether this system behaves in the Mega Man style where you can use the powers of one boss against another or in the flat style where you get a fixed amount of XP and the battle order does not matter.

The only thing remaining is the final question, namely what will happen once all six power glasses have been reunited? We have not actually thought that far ahead yet. Considering that all of this is stolen from Dragon Ball, probably something fairly anticlimactic.

Post scriptum

Please don't actually go out and fight these people and steal their glassware. It would be very inconvenient for everyone involved. 

[1] Not guaranteed to contain an actual volcano, or even a replica of same.

sunnuntai 10. helmikuuta 2019

Why is cross-language cooperation so hard?

My previous blog post on the future of Meson was a smashing hit (with up to five(!) replies on Twitter), so I figured I'd write a bit more in depth article about the issue. I'd like to emphasize that like previously, this article only mentions Rust by name, but the issues raised are identical for all new programming languages such as D, Go, Nim, and all the others whose names I don't even know.

Let's start with a digression.

Never use the phrase "why don't you just"!

Whenever you discuss on the Internet and some sort of major problem appears, someone will come up and say something along the lines of "well, why don't you just do X", usually as the very first response.

Please don't do that.

Using this phrase (or any variation thereof) makes the person saying seem to do one of two things:
  1. Trying to handwave the actual problem away by belittling it.
  2. Not understanding (or even wanting to understand) the problem and instead blindly pushing their own agenda.
Note that even if you are not doing this, other people will assume that you are. It does not really matter what you think or know or say, only how other people perceive your message. This is perfectly demonstrated in this clip from the movie Sneakers.



Using phrases like "why don't you just" make you look bad, even if your message and intentions were good.

Well why don't you just call Cargo then?

Why indeed. This is the most common reply I get when this issue arises. So let's dive deep to find out from first principles why this is. Usually when people say this they envision a project that looks something like this.


That is, there is some standalone piece of Rust code and then Meson adds something on top (or builds a completely different executable or something). The two are fairly well separated and doing their own things. For these kinds of projects having one call the other works fine. If this was the case we could all go home. But let's make the project slightly more complex.


This this case we have an executable and a library liba that are built with Meson. In addition there is a Rust library libb that also uses liba internally. This is where things get complicated.

The first thing to note is that in this case liba must only be built once and both the Rust library and the executable must use the same version of the library. Any solution that would require having two versions of liba are not acceptable. This means that there are two different interfaces between these build systems. The first is the injection interface by which Meson tells Cargo where and how it built liba. The second is the extraction interface by which Meson gets information from Cargo about what it built and how. Neither of these exist in a form that would be actually useful.

For extraction Cargo has an unstable feature called build-plan, which spits out the compilation commands Cargo would take. The idea is that the outer build system could take these commands, run them on their own and then things would work. But they don't because compilation arguments do not have all the information necessary. Suppose both these libraries are linked statically. Then linking against libb is not enough and build will fail, you also need to link against liba (and in a very specific order even). If the libraries are shared, then you should not add liba on the link line. The extraction interface would need to have a full serialisation of these dependencies. Otherwise the end result is not guaranteed to work and depends on random things such as whether the internal use of liba is before or after libb on the executable's link command line. Consuming of libraries does not work without this.

The injection interface is even trickier. Based on a quick review, many Rust libraries either build their deps themselves (which is bad) or call pkg-config (which is good if your deps come from the system, but in this case they don't). But to make this use case work there would be a need to tell from "the outside" how to obtain which library and what their dependencies are. As far as I know there is no way to tell this to Cargo and any project that does its own thing with build.rs is automatically guaranteed to not work without code changes.

And that was the easy part

Suppose that in one of the libraries generates headers during compilation. An example would be Protocol Buffers. You can't start compiling any code that depends on those headers until they exist.  You could try to work around this by building one project at a time. This is the model of recursive Make and Visual Studio solution files. Its main downside is that it makes builds a lot slower. The interface specification would need to have the header information to get reliable builds.

Another thing about Protocol Buffers is that you need to build the compiler executable in order to compile proto definitions to sources. Like with headers there needs to be a way to tell the project to use the recently built protoc rather than trying to find it on its own. This also adds a new dependency between the projects. The interface specification needs to have all this information to get reliable builds.

All of this also needs to be in the extraction interface, too, in case library b generated headers or has an executable that should be used later on in the build.

The interface seems way too complicated...

Going through all the requirements you eventually find that in order to make things really, actually, work you need to serialise pretty much everything between the two projects. It would also need to be fully programming language agnostic. If we started the design work today it would take months or even years to get something working. Fortunately we don't necessarily have to.

This is exactly what Meson is.

All the work we have put in Meson for the past six years has been done to create a programming language independent way of describing builds and making them work together. As a side effect we have also created a reference implementation for such a build system that can be used to build code. But that is mostly incidental, the actual important thing is, and always has been, the description language. Every problem mentioned in this post has a solution in the Meson language and they have been used in production for year. Of course there would be problems and bugs found as the usage would grow but it's the same with every program. Bugs can be fixed.

What would it look like?

The build definition for library b that would support all the things discussed above would look like this (assuming the library has a pre-existing .h file and does not have a build.rs file):

project('libb', 'rust')

a_dep = dependency('a', fallback: ['a', 'a_dep'])
b = static_library('b', 'b.rs', dependencies: a_dep)
b_dep = declare_dependency(link_with: b, include_directories: '.')

And that's it. This is all you need to get full project interop (there may be some implementation bugs).

Best of all this can be implemented in 100% Rust without adversely affecting existing projects. Either Cargo can parse the toml definitions and export this file, or Cargo could be made to read Meson build definitions natively. As an extra bonus any Rust project that uses external dependencies could use this to get native and transparent pkgconfig support without needing to write a build.rs (obviously it would not work with Cargo proper unless support is added to Cargo to read Meson files).

Extra language advantage

There are many languages that want to become the next big systems programming language. Almost all of them advertise good C compatibility as a big selling point. There is not a single one, though, that has "good interoperability with C tooling" in their priority list. In fact it is often quite the opposite. Getting language specific build systems working within existing projects can be a massive pain because they have not been designed for that.

Thus if a language would take tooling compatibility as a priority and would produce Meson files, either as their build system's native format or through some sort of an export mechanism, it would have an ease of integration advantage over other languages. Libraries written in language X could be dropped inside other projects that build with Meson and they would basically work out of the box (again, modulo bugs).

So is this the One True Way of how things will go?

No. Of course not. There are many different ways to solve this problem and this is only one possible way (in my opinion a fairly good one, but obviously other people will have different opinions and priorities). The real solution will, at least in the short term, probably be something completely different. But in any case it should fulfil the requirements discussed above because you can't really combine two languages and get a usable and reliable combination without them.

tiistai 5. helmikuuta 2019

On possible futures of Meson

At FOSDEM I talked to a bunch of people about an issue that has been brought up a couple of times recently, specifically that of integrating Rust in existing code bases. Some projects are looking into converting parts (or presumably eventually everything) of their code to Rust. A requirement of this is that for some time there need to be both Rust and C or whatever language within one project at the same time. (The rest of this blog post will use Rust as an example, but the same issues are present in all modern programming languages that have the same build system/dependency setup. In practice this means almost all of them.)

This would not be such a problem except that Rust by itself has pretty much nothing in the standard library and you need to get many crates via Cargo for even fairly simple programs. Several people do not seem particularly thrilled about this for obvious reasons, but have given up on this battle because "in practice it's impossible to develop in Rust without using Cargo" or words to that effect. As the maintainer of Meson, they obviously come to me with the integration problem. Meson does support compiling Rust directly, but it does not go through Cargo.

This is where I'm told to "just call Cargo" instead. There are two major problems with this. The first one is technical and has to do with the fact that having two different build systems and dependency managers in one build directory does not really work. We're not going to talk about this issue in this blog post, interested people can find writings about this issue using their favorite bingoogle. The second issue is non-technical, and the more serious one.

This approach means the death of Meson as a project

If we assume (as seems to be the case currently) that new programming languages such as Rust take over existing languages and that they all have their own build system, then Meson becomes unnecessary. Having a build system whose only task is to call a different build system is useless. It is nothing but bloat that should be deleted. As the person whose (unpaid) job it is to care about the long term viability of Meson, this does not make me feel particularly good.

So what might the future might in hold? Let's look at some alternatives.

New languages take over and keep their own build systems

New things take over, old code gets rewritten or thrown away. Each language keeps living in its own silo. Language X is better than language Y battles become the new Emacs vs vi battles. Meson becomes obsolete and gets shipped to the farm where it joins LISP machines, BeOS and Amigas as technology that was once interesting and nice but then was forgotten.

This is where things are currently heading. It is also the most likely outcome.

New languages adopt Meson as their build system

Rather than building and maintaining their own build systems, new languages switch to using Meson so everyone has a shared system and things work together. This will not happen. The previous statement is so important that I'll write it here for a second time, but this time in bold and on its own line:

This will not happen!

Every new language wants to provide a full toolchain and developer experience on their own. This is actually a good choice because it means that they can provide a full dev experience by themselves without depending on any external tooling (especially one that is written in a different programming language). Being able to e.g. just install one MSI package on Windows and have the full dev experience there is a really nice thing to have. Because of this no new programming language will accept an external build tool. No way. No how.

Meson becomes a glue layer to combine programming languages

A common suggested solution is that language native build systems export their state somehow so that external tools can drive them. Meson would then take these individual projects and make them work together. On the face of it this seems simple and workable and allows mixing languages freely. But if you think about it more, this is not a great place to be in. Basically it means that your project's purpose changes from being a build system to being responsible of all project integration but who does not have any say on how builds happen. There is also no leverage on the actual toolchain developers.

A condensed description of this task would be integration hell as a service. It is unlikely that anyone would want to spend their free time doing that. I sure as hell don't. I deal with enough of that stuff at work already.

So is everything terrible and all hope lost forever?

Maybe.

But then again maybe not.

Meson the language is not Meson the implementation

Meson was designed from the beginning to be programming language independent. This means both the languages it supports and the language it is implemented in. It does not leak the fact that it's implemented in Python to the build definition language. The core Meson functionality is only a few thousand lines of Python, reimplementing it is not a massive task. There is nothing wrong with having multiple implementations. In fact it is encouraged. Feel like rewriting it in Rust? Go for it (pun not intended).

In this approach new programming languages would get to keep their own toolchains, installs and build systems. The only thing that would change is the build description language. The transition would take a fair bit of work to be sure, but the benefit is that all that code becomes easily integratable with other languages. The reference implementation of language X could even choose to only support its own language and hand multi-language projects off to the reference implementation.

There are other advantages as well. For example currently (or the last time I looked, which admittably was some time ago) Cargo farms almost all non-standard work to build.rs, leading to wheel reinvention. Meson has native support for most of the things a project might need (source code generation etc). Thus each language would not need to design their own DSL for describing these, but could instead take Meson's and implement it in their own tools. Meson also has a lot of functionality for things like creating Python modules, which you would also get for free.

But the main advantage of all of this would be cooperation. You could mix and match languages with ease (assuming a working API/ABI, but that is outside the build system's purview) and switch between different languages without needing to learn yet another build definition language and toolchain behaviour.

The main downside of this approach is that it can not be injected from "the outside". It would only work if there are people inside the language communities that would consider this a worthwhile thing to do and would drive the change from within. Even for them it would probably be a long battle, because these sorts of changes have traditionally met with a lot of resistance.

How will all this affect Meson development?

In the short term probably not in any noticeable way.

And in the long term?


lauantai 2. helmikuuta 2019

Meson's new logo: a design process

At FOSDEM, quite literally only a few moments ago, we published the new Meson logo.



The design process was fairly long and somewhat arduous. This blog post aims to provide the rationale for all aspects of the design process.

The shape

From the very beginning it was decided that the logo's shape should be based on the Reuleaux triangle. The overall shape is smooth and pleasant, yet mathematically precise. In spite of its simplicity it can be used for surprisingly complex things. Perhaps the best known example is that if you have a drill bit shaped like a Reuleaux triangle, it can be used to create a rectangular hole. Those who have knowledge about compiler toolchains know that these sort of gymnastics are exactly what a build system needs to do. 

There were tens of different designs that tried to turn this basic idea into a logo. In practice it meant sitting in front of Inkscape and trying different things. These attempts included such things as combining the Reuleaux triangle with other shapes such as circles and triangles, as well as various attempts to build a stylished "M" letter. None of them really clicked until one day I put a smaller triangle upside down inside the other. Something about that shape was immediately captivating.



This shape was not directly usable by itself because it looked a lot like the Sierpinski triangle, which you may know from many corporate logos and other media. People also felt that this shape had an eye or a face that was looking at them, which was distracting. The final breakthrough came when even smaller versions of the triangle were added to the corners of the smaller triangle to produce a roughly propeller-like shape. 

The colour

Meson's original logo was green as it is the colour of growth and nature. This is a common choice for tooling. As an example, Bazel's original logo was a stylised green leaf. It is easy to see why green does not work with the chosen shape:



Legend of Zelda alliterations aside, some other color was needed. I was reading an article about logo colors which mentioned that for some reason purple logos are not as common as one would imagine. Upon reading that sentence the choice was done. After all if it is good enough for Taco Bell and Barbie, it is good enough for us.

Choosing the actual shade of purple was a tougher problem. We wanted a deep shade that is closer to blue than red. A bunch of back and forth got us to the current selection, PANTONE 2105C. This decision might not be final, though. The shade is slightly off from what I considered the perfect one. It would also be nice to have a free-as-in-freedom way of defining the logo colour rather than using one from a proprietary vendor.

The font

Technical logos usually use sans serif fonts, possibly with a geometrical shape to evoke stability and rigidity. This was the original plan for Meson's logo as well but eventually it seemed like those fonts were fairly heavy. We wanted the text to be light to symbolize the lean internals of Meson and its low resource usage. It also turned out that a light text nicely balances the slightly bulky logo shape. Going through various font and design sites I eventually found myself looking at this picture.


This is the inscription in Trajan's Column as created by ancient Romans. The typeface was exactly as needed, light but a definite sense of gravitas. The basic letterforms were traced and after several rounds of artisanal hand-crafted vector object tuning the final text shape was finished. For comparison we also tested Computer Modern as the logo text. It got a very sharp 50/50 split of people either hating it or loving it.

Licenses

The logo is not licensed under Apache as the rest of Meson is. It turns out that there is a license hole in the FOSS world for logos. It would be great if someone could come up with a simple and clear cut "you can use the logo to refer to the project but not claim endorsement of the project or use the logo on any merchandising" license. Many projects, such as GNOME, have their own license for this but copying random license legalese always feels a bit scary.

Final words

If you spend long enough working on any abstract geometric shape, you are going to think of some real world object that resembles it. That's just how the human brain works. This the image my subconscious brought forth.


I swear that this was not an intentional design point.