A nice feature provided by almost every Unix system is rpath. Put simply it is a way to tell the dynamic linker to look up shared libraries in custom directories. Build systems use it to be able to run programs directly from the build directory without needing to fiddle with file copying or environment variables. As is often the case, Windows does things completely differently and does not have a concept of rpath.
In Windows shared libraries are always looked up in directories that are in the current PATH. The only way to make the dynamic linker look up shared libraries in other directories is to add them to the PATH before running the program. There is also a way to create a manifest file that tells the loader to look up libraries in a special place but it is always a specially named subdirectory in the same directory as the executable. You can't specify an arbitrary path in the manifest, so the libraries need to be copied there. This makes Windows development even more inconvenient because you need to either fiddle with paths, copy shared libraries around or statically link everything (which is slooooow).
If you look at Windows executables with a hex editor, you find that they behave much the same way as their unixy counterparts. Each executable contains a list of dependency libraries that it needs, such as helper.dll. Presumably what happens is that at runtime the dynamic linker will parse the exe file and pass the library names to some lookup function that finds the actual libraries given the current PATH value. This raises the obvious question: what would happen if, somehow, the executable file would have an absolute path written in it rather than just the filename?
It turns out that it works and does what you would expect it to. The backend code accepts absolute paths and resolves them to the correct file without PATH lookups. With this we have a working rpath simulacrum. It's not really workable, though, since the VS toolchain does not support writing absolute paths to dependencies in output files. Editing the result files by hand is also a bit suspicious because there are many things that depend on offsets inside the file. Adding or removing even one byte will probably break something. The only thing we can really do is to replace one string with a different one with the same length.
This turns out to be the same problem that rpath entries have on Unix and the solution is also the same. We need to get a long enough string inside the output file and then we can replace it with a different string. If the replacement string is shorter, it can be padded with null bytes because the strings are treated as C strings. I have written a simple test repository doing this, which can be downloaded from Github.
On unix rpath is specified with a command line argument so it can be padded to arbitrary size. Windows does not support this so we need to fake it. The basic idea is simple. Instead of creating a library helper.dll we create a temporary library called aaaaaaaaaaaaaaaaaaaaaaaa.dll and link the program against that. When viewed in a hex editor the executable looks like this.
Now we can copy the library to its real name in a subdirectory and patch the executable. The result looks like this.
The final name was shorter than what we reserved so there are a bunch of zero bytes in the executable. This program can now be run and it will always resolve to the library that we specified. When the program is installed the entry can be changed to just plain helper.dll in the same way making it indistinguishable from libraries built without this trick (apart from the few extra null bytes).
Rpath on Windows: achieved.
Is this practical?
It's hard to say. I have not tested this on anything except toy programs but it does seem to work. It's unclear if this was the intended behaviour, but Microsoft does take backwards compatibility fairly seriously so one would expect it to keep working. The bigger problem is that the VS toolchain creates many other files, such as pdb debug info files, that probably don't like being renamed like this. These files are mostly undocumented so it's difficult to estimate how much work it would take to make binary hotpatching work reliably.
The best solution would be for Microsoft to add a new linker argument to their toolchain that would write dependency info to the files as absolute paths and to provide a program to rewrite those entries as discussed above. Apple already provides all of this functionality in their core toolchain. It would be nice for MS to do the same. This would simplify cross platform development because it would make all the major platforms behave in the same way.
On the Meson manual crowdfunding page it is mentioned that the end result can not be put under a fully free license. Several people have said that they "don't believe such a law could exist" or words to that effect. This blog post is an attempt to to explain the issue in English as all available text about the case is in Finnish. As a disclaimer: I'm not a lawyer, the following is not legal advice, there is no guarantee, even that any of the information below is factual.
To get started we need to go back in time a fair bit and look at disaster relief funds. In Finland you must obtain a permit from the police in order to gather money for general charitable causes. This permit has strict requirements. The idea is that you can't just start a fundraising, take people's money and pocket it, instead the money must provably go to the cause it was raised for. The way the law is written is that a donation to charity is done without getting "something tangible" in return. Roughly if you give someone money and get a physical item in return, it is considered a sales transaction. If you give money to someone and in return get a general feeling of making the world better in some way, that is considered a donation. The former is governed by laws of commerce, the latter by laws of charity fundraising.
A few years ago there was a project to create a book to teach people Swedish. The project is page is here, but it is all in Finnish so it's probably not useful to most readers. They had a crowdfunding project to finish the project with all the usual perks. One of the goals of the crowdfunding was to make the book freely distributable after publishing. This is not unlike funding feature work on FOSS projects works.
What happened next is that the police stepped in and declared this illegal (news story, in Finnish). Their interpretation was that participating in this campaign without getting something tangible in return (i.e. paying less than the amount needed to get the book) was a "charitable donation". Thus it needs a charity permit as explained above. Running a crowdfunding campaign is still legal if it is strictly about pre-sales. That is, every person buys "something" and that something needs to have "independent value" of some sort. If the outcome of a project is a PDF and that PDF becomes freely available, it can be argued that people who participated did not get any "tangible value" in exchange for their money.
Because of this the outcome of the Meson manual crowdfunding campaign can not be made freely available. This may seem a bit stupid, but sadly that's the law. The law is undergoing changes (see here, in Finnish), but those changes will not take effect for quite some time and even when they do it is unclear how those changes would affect these kinds of projects.
The Meson Build system has existed for over six years. Even though we have a fairly good set of documentation, there has not been a standalone user's manual for it. Until now.
A crowdfunding campaign to finance the manual has just been launched on Indiegogo. The basic deal is simple, for 30€ you get the final book as a PDF. To minimize work and save trees, there is no physical version. There are also no stickers, beer mats or any other tchotchkies. There are a few purchase options as well as opportunities for corporate sponsorships. Please see the Indiegogo project page for further details. If there are any questions about this campaign feel free to contact me. The easiest way is via email.
Overall I'm quite excited about this campaign. One reason is obviously personal, but the other has to do with sustainability of FOSS projects in general. There has been a lot of talk about how maintainers of open source projects can get compensated for their work. This campaign can be seen as an experiment to see if the crowdfunding model could work in practice.
So if you are just getting started with building software and want a user manual, buy this book. If you have basic experience with Meson and want to dive deeper, buy this book. If you are a seasoned veteran and don't really need a book but want to support the project (specifically me), buy this book. Regardless of anything else, please spread the word on your favourite social media and real world venues of choice.
Do not, I repeat, NOT make your test framework fail a test run if it writes any text to stderr! No matter how good of on idea you think it is, it's terrible.
If you absolutely, positively have to do that, then print the reason for this failure in your output log. If you can't think of a proper warning message, feel free to copy paste this one:
THIS TEST FAILED BECAUSE IT WROTE TO STDERR AND SOMEONE HERE (OBVIOUSLY NOT ME) THOUGHT MAKING THAT A HARD ERROR WOULD BE A GOOD IDEA!!!!!!!
Sincerely: a person who has lost hours of his life on this sh*t on multiple occasions and can never get it back.
It is fairly well established that Linux is not the #1 game development platform at this point in time. It is also unlikely to be the most popular one any time soon due to reasons of economics and inertia. A more interesting question, then, would be can it be made the best one? The one developers favour over all others? The one that is so smooth and where things work so well together that developers feel comfortable and productive in it? So productive that whenever they use some other environment, they get aggravated by being forced to use such a substandard platform?
Maybe.
The requirements
In this context we use "game development" as a catch all term for software development that has the following properties:
The code base is huge, millions or tens of millions lines of code.
Non-code assets are up to tens of gigabytes in size
A program, once built, needs to be tested on a bunch of various devices (for games, using different 3D graphics cards, amounts of memory, processors etc).
What we already have
A combination of regular Linux userland and Flatpak already provides a lot. Perhaps the most unique feature is that you can get the full source code of everything in the system all the way down to the graphic cards' device drivers (certain proprietary hardware vendors notwithstanding). There is no need to guess what is happening inside the graphics stack, you can just add breakpoints and step inside it with full debug info.
Linux as a platform is also faster than competing game development systems at most things. Process invocation is faster, file systems are faster, compilations are faster, installing dependencies is faster. These are the sorts of small gains that translate to better productivity and developer happiness.
Flatpak as a deployment mechanism is also really nice.
What needs to be improved?
Many tools in the Linux development chain assume (directly or indirectly) that doing something from scratch for every change is "good enough". Once you get into large enough scale this no longer works. As an example flatpak-builder builds its packages by copying the entire source tree inside the build container. If your repository is in the gigabyte range this does not work, but instead something like bind mounting should be used. (AFAIK GNOME Builder does something like this already.) Basically every operation needs to be O(delta) instead of O(whole_shebang):
Any built program that runs on the developer's machine must be immediately deployable on test machines without needing to do a rebuild on a centralised build server.
Code rebuilds must be minimal.
Installs must skip files that have not changed since the last install.
Package creation must only account for changed files.
All file transfers must be delta based. Flatpak already does this for package downloads but building the repo seems to take a while.
A simple rule of thumb for this is that changing one texture in a game and deploying the result on a remote machine should not take more than 2x the amount of time it would take to transfer the file directly over with scp.
Other tooling support
Obviously there needs to be native support for distributed builds. Either distcc, IceCream or something fancier, but even more important is great debugging support.
By default the system should store full debug info and corresponding source code. It should also log all core dumps. Pressing one button should then open up the core file in an IDE with up to date source code available and ready to debug. This same functionality should also be usable for debugging crashes in the field. No crash should go unstored (assuming that there are no privacy issues at play).
Perhaps the hardest part is the tooling for non-coders. It should be possible to create new game bundles with new assets without needing to touch any "dev" tools, even when running a proprietary OS. For example there could be a web service where you could do things like "create new game install on machine X and change model file Y with this uploaded file Z". Preferably this should be doable directly from the graphics application via some sort of a plugin.
Does something like this already exist?
Other platforms have some of these things built in and some can be added with third party products. There are probably various implementations of these ideas inside the closed doors of many current game development studios. AFAICT there does not exist a fully open product that would combine all of these in a single integrated whole. Creating that would take a fair bit of work, but once done we could say that the simplest way to set up the infrastructure to run a game studio is to get a bunch of hardware, open a terminal and type:
In interesting intellectual design challenge is to take a working thing (library, architecture, etc) and then see what would happen if you would reimplement it with the exact opposite way. Not because you'd use the end result anywhere, but just to see if you can learn something new. Or, in other words:
As an example let's apply this approach to C++'s fixed size array object or std::array. Some of its core design principles include:
Indexing is by default unsafe, user is responsible for providing valid indexes.
Errors are reported via exceptions.
Iterators may be invalid and invoking invalid iterators is allowed but UB.
Indexing must be exactly as fast as accessing a C array.
Inverting all of these give us the following design principles:
Out of bound accesses must be impossible.
No exceptions (assuming contained objects do not throw exceptions).
All iterator dereferences are guarded and may never lead to bad accesses.
It's ok to use some (but not much) processor time to ensure safety. Aim for zero overhead when possible.
So how does it look like?
An experimental PoC implementation can be found in this Github repo. Note that the code is intentionally unpolished. There are silly choices made. That is totally fine, it's not meant for actual use, only to explore the problem space.
The most important operation for an array type is indexing. It must work for all index values, even for those out of bounds. As no exceptions are allowed, the natural way to make this work is to return a Maybe type. This could be a std::optional<std::reference_wrapper<T>>, but for reasons that will become apparent later, we use a custom type. The outcome for this is kind of nice, allowing you to do things like (assuming an array of size 20):
assert(x[0]); // check if index is valid.
assert(!x[20]); // OOB indexes are invalid
*x[0] = 5 // assignment
assert(*x[0] == 5); // dereference
int i = *x[20]; // aborts program
The overall usability is kind of nice. This is similar to languages that have int? variables. The biggest problem here is that there is no way to prevent dereferencing an invalid maybe object, leading to termination. A typical bug would look like this:
if(maybe) {
*maybe = 3;
} else {
*maybe = 4; // Legal code but should not be.
}
There are at least three possible architectural ways to solve this:
Pattern matching (a switch/case on the object type) with language enforcement.
A language construct for "if attempting to use an invalid maybe object, exit the current function with an error". There have been talks of a try statement that would do something like this.
Maybes can not be dereferenced, only called with a method like visit(if_valid_function, if_not_valid_function).
#3 can be done today but is tedious and still does not permit automatically returning an error from the current function block.
Iteration
Creating a safe iterator is fairly simple. This iterator has a pointer to the original object and an integer offset. Dereferencing it calls the indexing operation and returns the maybe to the caller. This works fine until you test it with std::sort and after a lot of debugging find out that the implementation has a code block looking like this:
T tmp = std::move(a);
a = std::move(b);
b = std::move(tmp);
Even if you have swap defined for your objects, it will call this at some point. The problem here is that since the temporary does not point to any existing object, it can not store the value of the first move. There does not seem to be a good solution for this. Swap-only types might work, but such a type can not be defined with C++'s type system (or at least std::sort can not handle such types). The solution used in this example is that if a maybe object is created so that it does not point to any existing object, it stores objects inside itself. This works (for this use case) but feels a bit iffy.
Another problem this exposes is that the STL does not really work with these kinds of types. The comparison function returns a boolean, but if you compare (for whatever reason) invalid iterators and don't use exceptions there should be three different return values: true, false and erroneous_comparison. That is, an std::optional<bool>. Fixing this properly would mean changing std::sort to handle failing comparisons.
But what about performance?
For something like sorting, checking every access might cause a noticeable performance hit. If you do something like this:
std::sort(coll.begin(), coll.end())
it is fairly easy to verify that the range is valid and all thus accesses will be valid (assuming no bugs in the sort implementation). It would be nice to be able to opt out of range checks in these cases. Here's one way of doing it:
Here unsafe is a helper class that returns raw pointers to the underlying object. In this way individual accesses (which are bug prone) are guarded but larger batch operations (which are safer) can optionally request a faster code path. The unsafe request is immediately apparent and easily greppable.
Is this fully safe?
No. There are at least three different ways of getting an invalid reference.
Deleting the main object while iterators/maybes are outstanding. Would require tracking all outstanding objects. Probably too heavyweight. In Rust this is handled by the borrow checker at compile time.
Unallocating the memory backing the object without running its destructor. This is actually legal. Can't be guarded against.
Changing the object's backing memory to read only with a syscall. Can't be guarded against.
Most FOSS projects have a bug tracker, code repository and an IRC channel. Many even have a developer meeting, conference or awards given out to most prolific contributors. But do you know what no FOSS project has that they really should?
A Dragon Ball/NES/SNES style RPG quest.
To fill this void in the FOSS experience we have taken it upon ourselves to create one. The first thing any quest needs is artifacts worth fighting over. Going with the style of Dragon Ball we have created six power glasses infused with the power of software building. This is what they look like.
These items were hand-crafted in a blazing Finnish furnace in the dead of winter. Once cast and cooled the Meson logo and the developer team's secret motto were debossed on the side by artisanal free range engraving elves of Espoo.
To make the quest more challenging, we have scattered all the power glasses to the extreme ends of the world. Each is being held by a mini boss who must be defeated in order to obtain the corresponding power. In no particular order they are:
Igor Gnatenko, Czech republic, holding the power of the First Community Member
Nirbheek Chauhan, India, holding the power of the Co-Maintainer
Dylan Baker, USA, holding the power of 3D Graphics
Alexis Jeandet, France, holding the power of CI
Aleksey Filippov, Ireland, holding the power of Dependency Packaging
They have all stood as guardians of their respective power for an extended amount of time and for that they deserve vast heaps of praise and gratitude from all of us.
The sixth and last power glass is held under strict supervision in my Finnish volcano lair headquarters [1] and can only be unlocked when holding all the other five power glasses.
It is currently unclear whether this system behaves in the Mega Man style where you can use the powers of one boss against another or in the flat style where you get a fixed amount of XP and the battle order does not matter.
The only thing remaining is the final question, namely what will happen once all six power glasses have been reunited? We have not actually thought that far ahead yet. Considering that all of this is stolen from Dragon Ball, probably something fairly anticlimactic.
Post scriptum
Please don't actually go out and fight these people and steal their glassware. It would be very inconvenient for everyone involved.
[1] Not guaranteed to contain an actual volcano, or even a replica of same.
My previous blog post on the future of Meson was a smashing hit (with up to five(!) replies on Twitter), so I figured I'd write a bit more in depth article about the issue. I'd like to emphasize that like previously, this article only mentions Rust by name, but the issues raised are identical for all new programming languages such as D, Go, Nim, and all the others whose names I don't even know.
Let's start with a digression.
Never use the phrase "why don't you just"!
Whenever you discuss on the Internet and some sort of major problem appears, someone will come up and say something along the lines of "well, why don't you just do X", usually as the very first response.
Please don't do that.
Using this phrase (or any variation thereof) makes the person saying seem to do one of two things:
Trying to handwave the actual problem away by belittling it.
Not understanding (or even wanting to understand) the problem and instead blindly pushing their own agenda.
Note that even if you are not doing this, other people will assume that you are. It does not really matter what you think or know or say, only how other people perceive your message. This is perfectly demonstrated in this clip from the movie Sneakers.
Using phrases like "why don't you just" make you look bad, even if your message and intentions were good.
Well why don't you just call Cargo then?
Why indeed. This is the most common reply I get when this issue arises. So let's dive deep to find out from first principles why this is. Usually when people say this they envision a project that looks something like this.
That is, there is some standalone piece of Rust code and then Meson adds something on top (or builds a completely different executable or something). The two are fairly well separated and doing their own things. For these kinds of projects having one call the other works fine. If this was the case we could all go home. But let's make the project slightly more complex.
This this case we have an executable and a library liba that are built with Meson. In addition there is a Rust library libb that also uses liba internally. This is where things get complicated.
The first thing to note is that in this case liba must onlybe built once and both the Rust library and the executable must use the same version of the library. Any solution that would require having two versions of liba are not acceptable. This means that there are two different interfaces between these build systems. The first is the injection interface by which Meson tells Cargo where and how it built liba. The second is the extraction interface by which Meson gets information from Cargo about what it built and how. Neither of these exist in a form that would be actually useful.
For extraction Cargo has an unstable feature called build-plan, which spits out the compilation commands Cargo would take. The idea is that the outer build system could take these commands, run them on their own and then things would work. But they don't because compilation arguments do not have all the information necessary. Suppose both these libraries are linked statically. Then linking against libb is not enough and build will fail, you also need to link against liba (and in a very specific order even). If the libraries are shared, then you should not add liba on the link line. The extraction interface would need to have a full serialisation of these dependencies. Otherwise the end result is not guaranteed to work and depends on random things such as whether the internal use of liba is before or after libb on the executable's link command line. Consuming of libraries does not work without this.
The injection interface is even trickier. Based on a quick review, many Rust libraries either build their deps themselves (which is bad) or call pkg-config (which is good if your deps come from the system, but in this case they don't). But to make this use case work there would be a need to tell from "the outside" how to obtain which library and what their dependencies are. As far as I know there is no way to tell this to Cargo and any project that does its own thing with build.rs is automatically guaranteed to not work without code changes.
And that was the easy part
Suppose that in one of the libraries generates headers during compilation. An example would be Protocol Buffers. You can't start compiling any code that depends on those headers until they exist. You could try to work around this by building one project at a time. This is the model of recursive Make and Visual Studio solution files. Its main downside is that it makes builds a lot slower. The interface specification would need to have the header information to get reliable builds.
Another thing about Protocol Buffers is that you need to build the compiler executable in order to compile proto definitions to sources. Like with headers there needs to be a way to tell the project to use the recently built protoc rather than trying to find it on its own. This also adds a new dependency between the projects. The interface specification needs to have all this information to get reliable builds.
All of this also needs to be in the extraction interface, too, in case library b generated headers or has an executable that should be used later on in the build.
The interface seems way too complicated...
Going through all the requirements you eventually find that in order to make things really, actually, work you need to serialise pretty much everything between the two projects. It would also need to be fully programming language agnostic. If we started the design work today it would take months or even years to get something working. Fortunately we don't necessarily have to.
This is exactly what Meson is.
All the work we have put in Meson for the past six years has been done to create a programming language independent way of describing builds and making them work together. As a side effect we have also created a reference implementation for such a build system that can be used to build code. But that is mostly incidental, the actual important thing is, and always has been, the description language. Every problem mentioned in this post has a solution in the Meson language and they have been used in production for year. Of course there would be problems and bugs found as the usage would grow but it's the same with every program. Bugs can be fixed.
What would it look like?
The build definition for library b that would support all the things discussed above would look like this (assuming the library has a pre-existing .h file and does not have a build.rs file):
project('libb', 'rust')
a_dep = dependency('a', fallback: ['a', 'a_dep'])
b = static_library('b', 'b.rs', dependencies: a_dep)
And that's it. This is all you need to get full project interop (there may be some implementation bugs).
Best of all this can be implemented in 100% Rust without adversely affecting existing projects. Either Cargo can parse the toml definitions and export this file, or Cargo could be made to read Meson build definitions natively. As an extra bonus any Rust project that uses external dependencies could use this to get native and transparent pkgconfig support without needing to write a build.rs (obviously it would not work with Cargo proper unless support is added to Cargo to read Meson files).
Extra language advantage
There are many languages that want to become the next big systems programming language. Almost all of them advertise good C compatibility as a big selling point. There is not a single one, though, that has "good interoperability with C tooling" in their priority list. In fact it is often quite the opposite. Getting language specific build systems working within existing projects can be a massive pain because they have not been designed for that.
Thus if a language would take tooling compatibility as a priority and would produce Meson files, either as their build system's native format or through some sort of an export mechanism, it would have an ease of integration advantage over other languages. Libraries written in language X could be dropped inside other projects that build with Meson and they would basically work out of the box (again, modulo bugs).
So is this the One True Way of how things will go?
No. Of course not. There are many different ways to solve this problem and this is only one possible way (in my opinion a fairly good one, but obviously other people will have different opinions and priorities). The real solution will, at least in the short term, probably be something completely different. But in any case it should fulfil the requirements discussed above because you can't really combine two languages and get a usable and reliable combination without them.
At FOSDEM I talked to a bunch of people about an issue that has been brought up a couple of times recently, specifically that of integrating Rust in existing code bases. Some projects are looking into converting parts (or presumably eventually everything) of their code to Rust. A requirement of this is that for some time there need to be both Rust and C or whatever language within one project at the same time. (The rest of this blog post will use Rust as an example, but the same issues are present in all modern programming languages that have the same build system/dependency setup. In practice this means almost all of them.)
This would not be such a problem except that Rust by itself has pretty much nothing in the standard library and you need to get many crates via Cargo for even fairly simple programs. Several people do not seem particularly thrilled about this for obvious reasons, but have given up on this battle because "in practice it's impossible to develop in Rust without using Cargo" or words to that effect. As the maintainer of Meson, they obviously come to me with the integration problem. Meson does support compiling Rust directly, but it does not go through Cargo.
This is where I'm told to "just call Cargo" instead. There are two major problems with this. The first one is technical and has to do with the fact that having two different build systems and dependency managers in one build directory does not really work. We're not going to talk about this issue in this blog post, interested people can find writings about this issue using their favorite bingoogle. The second issue is non-technical, and the more serious one.
This approach means the death of Meson as a project
If we assume (as seems to be the case currently) that new programming languages such as Rust take over existing languages and that they all have their own build system, then Meson becomes unnecessary. Having a build system whose only task is to call a different build system is useless. It is nothing but bloat that should be deleted. As the person whose (unpaid) job it is to care about the long term viability of Meson, this does not make me feel particularly good.
So what might the future might in hold? Let's look at some alternatives.
New languages take over and keep their own build systems
New things take over, old code gets rewritten or thrown away. Each language keeps living in its own silo. Language X is better than language Y battles become the new Emacs vs vi battles. Meson becomes obsolete and gets shipped to the farm where it joins LISP machines, BeOS and Amigas as technology that was once interesting and nice but then was forgotten.
This is where things are currently heading. It is also the most likely outcome.
New languages adopt Meson as their build system
Rather than building and maintaining their own build systems, new languages switch to using Meson so everyone has a shared system and things work together. This will not happen. The previous statement is so important that I'll write it here for a second time, but this time in bold and on its own line:
This will not happen!
Every new language wants to provide a full toolchain and developer experience on their own. This is actually a good choice because it means that they can provide a full dev experience by themselves without depending on any external tooling (especially one that is written in a different programming language). Being able to e.g. just install one MSI package on Windows and have the full dev experience there is a really nice thing to have. Because of this no new programming language will accept an external build tool. No way. No how.
Meson becomes a glue layer to combine programming languages
A common suggested solution is that language native build systems export their state somehow so that external tools can drive them. Meson would then take these individual projects and make them work together. On the face of it this seems simple and workable and allows mixing languages freely. But if you think about it more, this is not a great place to be in. Basically it means that your project's purpose changes from being a build system to being responsible of all project integration but who does not have any say on how builds happen. There is also no leverage on the actual toolchain developers.
A condensed description of this task would be integration hell as a service. It is unlikely that anyone would want to spend their free time doing that. I sure as hell don't. I deal with enough of that stuff at work already.
So is everything terrible and all hope lost forever?
Maybe.
But then again maybe not.
Meson the language is not Meson the implementation
Meson was designed from the beginning to be programming language independent. This means both the languages it supports and the language it is implemented in. It does not leak the fact that it's implemented in Python to the build definition language. The core Meson functionality is only a few thousand lines of Python, reimplementing it is not a massive task. There is nothing wrong with having multiple implementations. In fact it is encouraged. Feel like rewriting it in Rust? Go for it (pun not intended).
In this approach new programming languages would get to keep their own toolchains, installs and build systems. The only thing that would change is the build description language. The transition would take a fair bit of work to be sure, but the benefit is that all that code becomes easily integratable with other languages. The reference implementation of language X could even choose to only support its own language and hand multi-language projects off to the reference implementation.
There are other advantages as well. For example currently (or the last time I looked, which admittably was some time ago) Cargo farms almost all non-standard work to build.rs, leading to wheel reinvention. Meson has native support for most of the things a project might need (source code generation etc). Thus each language would not need to design their own DSL for describing these, but could instead take Meson's and implement it in their own tools. Meson also has a lot of functionality for things like creating Python modules, which you would also get for free.
But the main advantage of all of this would be cooperation. You could mix and match languages with ease (assuming a working API/ABI, but that is outside the build system's purview) and switch between different languages without needing to learn yet another build definition language and toolchain behaviour.
The main downside of this approach is that it can not be injected from "the outside". It would only work if there are people inside the language communities that would consider this a worthwhile thing to do and would drive the change from within. Even for them it would probably be a long battle, because these sorts of changes have traditionally met with a lot of resistance.
How will all this affect Meson development?
In the short term probably not in any noticeable way.
At FOSDEM, quite literally only a few moments ago, we published the new Meson logo.
The design process was fairly long and somewhat arduous. This blog post aims to provide the rationale for all aspects of the design process.
The shape
From the very beginning it was decided that the logo's shape should be based on the Reuleaux triangle. The overall shape is smooth and pleasant, yet mathematically precise. In spite of its simplicity it can be used for surprisingly complex things. Perhaps the best known example is that if you have a drill bit shaped like a Reuleaux triangle, it can be used to create a rectangular hole. Those who have knowledge about compiler toolchains know that these sort of gymnastics are exactly what a build system needs to do.
There were tens of different designs that tried to turn this basic idea into a logo. In practice it meant sitting in front of Inkscape and trying different things. These attempts included such things as combining the Reuleaux triangle with other shapes such as circles and triangles, as well as various attempts to build a stylished "M" letter. None of them really clicked until one day I put a smaller triangle upside down inside the other. Something about that shape was immediately captivating.
This shape was not directly usable by itself because it looked a lot like the Sierpinski triangle, which you may know from many corporate logos and other media. People also felt that this shape had an eye or a face that was looking at them, which was distracting. The final breakthrough came when even smaller versions of the triangle were added to the corners of the smaller triangle to produce a roughly propeller-like shape.
The colour
Meson's original logo was green as it is the colour of growth and nature. This is a common choice for tooling. As an example, Bazel's original logo was a stylised green leaf. It is easy to see why green does not work with the chosen shape:
Legend of Zelda alliterations aside, some other color was needed. I was reading an article about logo colors which mentioned that for some reason purple logos are not as common as one would imagine. Upon reading that sentence the choice was done. After all if it is good enough for Taco Bell and Barbie, it is good enough for us.
Choosing the actual shade of purple was a tougher problem. We wanted a deep shade that is closer to blue than red. A bunch of back and forth got us to the current selection, PANTONE 2105C. This decision might not be final, though. The shade is slightly off from what I considered the perfect one. It would also be nice to have a free-as-in-freedom way of defining the logo colour rather than using one from a proprietary vendor.
The font
Technical logos usually use sans serif fonts, possibly with a geometrical shape to evoke stability and rigidity. This was the original plan for Meson's logo as well but eventually it seemed like those fonts were fairly heavy. We wanted the text to be light to symbolize the lean internals of Meson and its low resource usage. It also turned out that a light text nicely balances the slightly bulky logo shape. Going through various font and design sites I eventually found myself looking at this picture.
This is the inscription in Trajan's Column as created by ancient Romans. The typeface was exactly as needed, light but a definite sense of gravitas. The basic letterforms were traced and after several rounds of artisanal hand-crafted vector object tuning the final text shape was finished. For comparison we also tested Computer Modern as the logo text. It got a very sharp 50/50 split of people either hating it or loving it.
Licenses
The logo is not licensed under Apache as the rest of Meson is. It turns out that there is a license hole in the FOSS world for logos. It would be great if someone could come up with a simple and clear cut "you can use the logo to refer to the project but not claim endorsement of the project or use the logo on any merchandising" license. Many projects, such as GNOME, have their own license for this but copying random license legalese always feels a bit scary.
Final words
If you spend long enough working on any abstract geometric shape, you are going to think of some real world object that resembles it. That's just how the human brain works. This the image my subconscious brought forth.
I swear that this was not an intentional design point.
When you create a new variable (in C, C++ and other languages) or allocate a block of memory the value is undefined. That is, whatever bit pattern happened to be in the raw memory location at the time. This is faster than initialising all memory (which languages such as Java do) but it is also unsafe and can lead to bugs, such as use-after-free issues.
There have been several attempts to change this behaviour and require that compilers would initialize all memory to a known value, usually zero. This is always rejected with a statement like "that would cause a performance degradation fo unknown size" and the issue is dropped. This is not very scientific so let's see if we could get at least some sort of a measurement for this.
The method
The overhead for uninitialized variables is actually fairly difficult to measure. Compilers don't provide a flag to initialize all variables to zero. Thus measuring this would require compiler hacking, which is a ton of work. An alternative would be to write a clang-tidy plugin and add a default initialization to zero for all variables that don't have a initialization clause already. This is also fairly involved, so let's not do this.
The impact of dynamic memory turns out to be fairly straightforward to measure. All we need to do is to build a shared library with custom overrides for malloc, free and memalign, and LD_PRELOAD it to any process we want to measure. The sample code can be found in this Github repo.
Measurements
We did two measurements. The first one was running Python's pystone benchmark. There was no noticeable difference between zero initialization and no initialization.
The second measurement consisted of compiling a simple C++ iostream helloworld application with optimizations enabled. The results for this experiment were a lot more interesting. Zeroing all memory on malloc made the program 2% slower. Zeroing the memory on both allocation and free (to catch use-after-free bugs) made the program 3.6% slower.
A memory zeroing implementation inside malloc would probably have a smaller overhead, because there are cases where you don't need to explicitly overwrite the memory, for example when the allocation is done behind the scenes via mmap/munmap.
At work I have to compile a large code base from scratch fairly often. One of the components it has is a 3D graphics library. It takes around 2 minutes 15 seconds to compile using an 8 core i7. After a while I got bored with this and converted the system to use a unity build. In all simplicity what that means is that if you have a target consisting of files foo.cpp, bar.cpp, baz.cpp etc you create a cpp file with the following contents:
#include<foo.cpp>
#include<bar.cpp>
#include<baz.cpp>
Then you would tell the build system to build that instead of the individual files. With this method the compile time dropped down to 1m 50s which does not seem like that much of a gain but the compilation used only one CPU core. The remaining 7 are free for other work. If the project had 8 targets of roughly the same size, building them incrementally would take 18 minutes. With unity builds they would take the exact same 1m 50s assuming perfect parallelisation, which happens fairly often in practice.
Wait, what? How is this even?
The main reason that C++ compiles slowly has to do with headers. Merely including a few headers in the standard library brings in tens or hundreds of thousands of lines of code that must be parsed, verified, converted to an AST and codegenerated in every translation unit. This is extremely wasteful especially given that most of that work is not used but is instead thrown away.
With an Unity build every #include is processed only once regardless of how many times it is used in the component source files.
Basically this amounts to a caching problem, which is one of the two really hard problems in computer science in addition to naming things and off by one errors.
Why is this not used by everybody then?
There are several downsides and problems. You can't take any old codebase and compile it as a unity build. The first blocker is that things inside source files leak into other ones since they are all textually included one after the other.. For example if you have two files and each of them declares a static function with the same name, it will lead to name clashes and a compilation failure. Similarly things like using namespace std declarations leak from one file to another causing havoc.
But perhaps the biggest problem is that every recompilation takes the same time. An incremental rebuild where one file has changed takes a few seconds or so whereas a unity builds takes the full 1m 50s every time. This is a major roadblock to iterative development and the main reason unity builds are not widely used.
A possible workflow with Meson
For simplicity let's assume that we have a project that builds and works with unity builds. Meson has an automatic unity build file generator that can be enabled by setting the value of the unity build option.
This solves the basic build problem but not the incremental one. However usually you'd develop only one target (be it a library, executable or module) and want to build only that one incrementally and everything else as a unity build. This can be done by editing the build definition of the target in question and adding an override option:
Once you are done you can remove the override from the build file to return everything back to normal.
How does this tie in with C++ modules?
Directly? Not in any way really. However one of the stated advantages of modules has always been faster build times. There are a few module implementations but there is very little public data on how they behave with real world codebases. During a CppCon presentation on modules Google's Chandler Carruth mentioned that in Google's code base modules resulted in 30% build time reduction.
It was not mentioned whether Google uses unity builds internally but they almost certainly don't (based on things such as this bug report on Bazel). If we assume that theirs is the fastest existing "classical" C++ build mechanism, which it probably is, the conclusion is that it is an order of magnitude slower than a unity build on the same source files. A similar performance gap would probably not be tolerated in any other part of the C++ ecosystem.
A recent trend in language design and devops deployment has been to not use shared libraries. Instead every application is rebuilt and statically linked for maximum performance. This is highly convenient in many cases. Some people even go as far as to declare shared linking, and with it any ABI stability, a dead relic of the past that is only unnecessary but actively harmful because maintaining ABI stability slows down language changes and renewal.
This blog post was not written to argue whether this is true or not. Instead it is meant to list many reasons and use cases where shared libraries and ABI stability are useful and which would be hard, or even impossible, to achieve by relying only on static linking.
Many of the issues listed here are written from the perspective of a modern Linux distribution, especially Debian. However I am not a Debian developer so the following is not any sort of an official statement, just my writings as an individual.
Guaranteed update propagation
Debian consists of thousands of packages. Each package's state is managed by a package maintainer. Each manager typically maintains between one and a handful of packages, so there are hundreds of them. Each one of them works in relative isolation from others. That is, they can upload updates to packages at their own pace. In fact, it is an important part of Debian's social structure that no-one can be forced to do any particular task.
On the other hand, Debian is also very strict about security. If a vulnerability is found in, say, a popular encryption library then it must be possible for one single person to update the encryption code in every single package that uses it, even indirectly. With a stable ABI and shared libraries, this can be done easily. Updating the dependency package (and possibly rebooting the machine) guarantees that every package on the system uses the new library. If packages were statically linked, each package would have to be rebuilt and reuploaded. This would require hundreds of people around the world to work in a coordinated fashion. In a volunteer based system this is not possible, especially for cases that require an embargo.
Update server bandwidth savings
The amount of bandwidth it takes to run a Linux distribution mirror is substantive. As we saw above, it is possible to update single packages which make downloads fairly small. If everything was statically linked then every library update would mean downloading the full rebuilt binaries of every affected package. This means a 10x to 100x increase in bandwidth requirements. Distro mirrors are already quite heavily loaded and probably could not handle this sort of increase in traffic.
Download bandwidth savings
Most of the population in the world does not have a direct 10GB Ethernet connection for their personal use. In fact there are many people who only have 2G connection at best and even that is sporadic. There are also many servers that have very poor Internet connections, such as scientific instruments and credit card payment terminals in remote cities. Getting updates to these machines is difficult even now. If update sizes ballooned in size, it might become completely infeasible.
Shipping prebuilt middleware
There are many providers of middleware (such as in computer games) that will only provide their code as prebuilt libraries (usually shared, because they are harder to reverse engineer). They will not and can not ever ship their source code to customers because that contains all their special sauce. This entire business model relies on a stable ABI.
Software certification
I don't know have personal experience about this so the following entry might be completely false. However it is based on best effort information I had. If you have first hand experience and can either confirm or deny this, please post a comment to this article.
In highly regulated business sectors the problem of certification often comes up. Basically what this means is that each executable is put through extensive testing cycle. If it passes then it is certified and can be used in production. Specifically, only that exact binary can be used. Any changes to the code means that the program must be re-certified. This is a time consuming and extremely expensive process.
It may be that the certification cycle is different for the operating system component. Thus applying OS updates provided by the vendor may be faster and cheaper. As long as they maintain ABI stability, the actual program does not need to be changed removing the need to re-certify it.
Extension modules
Suppose you create a program that provides an extension or plugin interface to third party code. Examples include the modding interface of many games and, as an extreme example, the entire Eclipse IDE. Supporting this without needing to provide third party extensions as source (and shipping a compiler with your program) requires a stable ABI.
Low barrier to entry
One of the main downsides of rebuilding everything from source all the time is the amount of resources it takes. For many this is not a problem and when asked about it may even snootily reply with "just buy more machines from AWS".
One of the strong motivations of the free and open source movement has been enablement and empowering. That is, making it as easy as possible for as many people as possible to participate. There are many people in the world whose only computer is an old laptop or possibly even just a Raspberry Pi. In the current model it is possible for take any part of the system and hack on it in isolation (except maybe something like Chromium). If we go to a future where participating in software development requires access to a data center, these people are prevented from contributing.
Supporting slow platforms
One of the main philosophical points of Debian is that every supported architecture must be self hosting. That is, packages for Arm must be built on Arm, Mips packages must be built on Mips and so on. Self hosting is an important goal, because it proves the system works and is self-sustaining in ways that simply using cross built packages does not.
Currently it takes a lot of time to do a full archive rebuild using any of the slower architectures, but it is still feasible. If the amount of work needed to do a full rebuild grows by 10 or 100, it is no longer achievable. Thus the only platforms that could reasonably self-host would be x86, Power, s390x and possibly arm64.
Supporting old binaries
There are many cases where a specific application binary must keep running even though the entire system around it changes. A good example of this are computer and console games. People have paid good money for games on Windows 7 (or Vista, or XP) and they expect them to keep working on Windows 10 as well, even on hardware that did not even exist back when the game was released. The only known solution to this are stable ABIs. The same problem happens with consoles such as PS4. Every single game released during its life cycle must run on all console system software versions released after the game, even without a network connection for downloading updates.
Errata
Since writing this article I have been told that any Developer may request a rebuild and reupload of a binary package and it happens automatically. So it is possible for one person to fix a package and have its dependents rebuilt, but it would still require lots of compute and bandwidth resources.
A few weeks ago I was at CppCon. One of the presentations was about new stuff in the Visual Studio compiler. The presentation had this slide fairly early on.
If that is truly their goal, then here are some things they could do. (Some not specifically about C++ but still related.)
Proper RPATH support
If you have a project that uses shared libraries and you want to run it directly from the build directory, then you really need to have rpath or something similar to it. A simple way of explaining it is that you add a piece of text inside an executable saying "when running me, search for foo.dll in directory ../baz/lib.
Since this is not natively supported, people need to resort to awful hacks to make it work:
adjusting the PATH envvar to contain the dirs where the dlls are (because PATH is used to look up dlls)
copy all files to the same directory before running
creating a manifest file defining an internal bundle, creating a subdirectory and copying all dependency dlls there
static linking everything
mandating a project layout where everything is in one subdirectory
All of these are nasty hacks. It should be possible to run programs straight from the build dir without needing to copy anything or change envvars.
During CppCon I was told that with the very latest Windows 10 it should be possible to do this "somehow" but googling for this has not uncovered any instructions.
Spawning a process using an array
The way to spawn processes in Windows is using the CreateProcess function. Note that it takes a command string, not an array. The command implementation will then parse the string to a command array and run it. The documentation page does not document how the parsing is done, but presumably it is the same as what cmd.exe does.
What this means is that it is impossible to spawn a process on Windows without needing to jump through massive quoting hoops. For example suppose you want to write a Ninja file to call a specific command. Because of this problem Ninja does not support arrays natively but instead requires every user to write a single command string, which leads to double quoting. First you need to quote the array to be a Windows process spawning command and then you need to quote that according to Ninja's quoting rules.
And then it gets terrible.
The command line length limitation on Windows is ridiculously short. Even fairly simple link commands are too long. Thus you need to write the actual command to a response file, which the command then reads and parses on its own. Since every program writes their own parsing and splitting code, you may find that you need to quote things differently depending on whether you are using the command line or a response file. You get one guess whether some (but not all) programs coming from Unix parse their response files according to Unix shell rules, even on Windows.
Now there might be a few people out there who just got outraged, because msvcrt does in fact have functions to spawn processes with arrays. They are a complete lie. Here is a rough pseudocode representation on how they are implemented:
def spawn_process(command_array):
command_line = ' '.join(command_array)
return CreateProcess(command_line)
Support GCC's destructor extension in plain C
RAII is awesome. It is, in fact, so awesome that GCC ships an extension to use it with plain C. It is used by many plain C projects such as GLib and systemd. I have spoken to many C developers and they really love that feature and they absolutely hate that they can't use it in code that has to support MSVC.
Adding this support would be great and make the world a better place in several ways including:
you can use libraries that use this feature as dependencies when building with MSVC
multiplatform projects can start using destructors freely
all the millions of lines of C code that exist in the world (and which will not be rewritten any time soon) can be made iteratively safer and more reliable
Eventually it would be nice to get this feature in the C standard, but that is unlikely to happen any time soon.
Performance optimize MSBuild
Running the test suite of Meson with the Visual Studio compiler takes roughly 6-7 minutes when using the Ninja backend and 14-18 minutes when using the MSBuild backend. Granted, this is a worst case scenario of running many small independent builds in a row, but it is still frustratingly slow. The same can be found when using Visual Studio IDE. After typing ctrl-shift-b there is usually a noticeable lag until any compilation actually starts.
Kill the need for vcvarsall.bat and provide parallel installable compilers
Visual studio compilers are not in path by default. You have to either start a special shell or run a magic bat file from a magic directory that sets up the environment so that the compilers work. If you go looking in the installed directory there are many different directories all of which contain an executable cl.exe. Which one you run depends on PATH settings, thus you can only run one compiler at a time. This makes it really difficult to, for example, run multiple different VS versions (15, 17, native, cross etc) from a single script.
This same problem has been solved on Unix side ages ago. The trick is to provide many executables with different names. For example cl15-x86.exe, cl17-arm.exe and cl17-x64.exe. Each of these executables would set up the equivalent of vcvarsall.bat for its own process and then forward the actual compilation to the compiler, wherever it may be hidden in the file system hierarchy. These binaries could the be put in one single path location and they could be used from any command prompt, even in parallel. This is particularly useful for cross compilation projects where you need to build a code generator with the native compiler and then use it to generate source code for the cross compiler.
Have you reported these as bugs upstream?
No. Nothing on this blog post is new, these are all issues that have been known for 20+ years and most likely have been reported to Microsoft dozens, if not hundreds of times. The fact that these things have not been fixed is a question of corporate priorities. As a random-non-windows-using-dude-on-internet I don't really have any influence on those.
Since times immemorial, compilers have been run as standalone batch processes. If you have 50 files to compile, then you invoke the compiler 50 times, once on each file. Since each compilation is independent of all others, the work can be parallelised perfectly. This seems like a simple and optimal solution.
But, as is commonly the case, this is not the whole truth. When compiling code, there are many subtasks that are common to each individual compilation and this causes a lot of duplication of effort. Perhaps the best known case of this are C++ templates. They are parsed and codegenerated for each file that uses them yielding in the same code in dozens of files. Then the linker comes along and throws all but one of them away. There are a bunch of other issues which are discussed in this video from LLVM developer's conference:
A problem of state preservation
One of the best known solution to this problem are precompiled headers. They work roughly like this:
Parse the contents of headers
Dump compiler internal state to a file
Load the file on each compiler invocation
The two main problems with this is that it requires someone to design and implement a full serialisation format for the compiler-internal data. That is a lot of tedious work that very few people will volunteer to do. The other downside is that the files need to be loaded explicitly from disk in every compilation process, which takes time, and that the build system needs to tell the compiler how to get this done. The granularity is also fairly coarse.
Ideally we would like to preserve as much data between two compiler invocations as possible without needing to serialise it to disk. As discussed in the above video, one solution is to have a "compiler plugin".
Almost every build system currently works roughly like this:
Read build definition (such as a Ninja file)
For each compilation, spawn a new compiler process and invoke the compiler executable
Shutdown
The proposed new model would go like this (no build system currently supports this, but adding it to e.g. Ninja is not a massive undertaking):
Read build definition
dlopen the compiler shared library file
For each compilation, create a new compiler object and invoke compilation using e.g. a thread pool
Destroy compiler objects and dclose the file
Shutdown
In this model all compilation jobs live in the same process, thus they can coordinate work behind the scenes however they wish. This requires some tricky code with thread safe caches and the like but it all internal to the compiler and never exposed. Even without caching this makes a difference on platforms such as Windows where process spawning is slow.
The big question remaining here is the API to use. It should have the following requirements:
Must be ABI stable in the C sense
Must be supportable on all compilers for all languages
Must expose the full functionality of the compiler
Must support an arbitrary number of compiler tasks within a single process
An API proposal for compiler invocation
On the face of it this seems like an impossible task. The API surface of a compiler is enormous and differs from compiler to compiler. However all of them already expose a stable ABI: the command line argument arrays. Exploiting this allows us to create an API supporting all of the requirements above with only six functions.
First we initialise the library:
CompilerService* compiler_init_service();
Here CompilerService is an opaque struct to a state object. There is one of these per process and it holds (internally) all the cached state and related things. Then we create a compiler object, one per compilation task:
CompilationResult* compiler_compile(Compiler *c, int argc, const char **argv);
This invocation matches the signature of the main function. Since we are not going through the shell/kernel we can pass an arbitrary number of arguments without needing to use response files, quote shell characters or any other nastiness. The return value contains the return code and the strings for stdout and stderr. The standalone compiler executable such as cl.exe could (in theory ;-) be implemented by just calling these functions and returning the results to the calling process.
The last thing we need are the deallocation functions:
When will this be available in <my favorite compiler>?
Probably not soon, this is all slideware. There is no actual code to implement this (that I know of at least). The big problem here is that most compilers have not been written with this sort of usage in mind. The have global variables and other things hostile to usage as a shared library. Fixing all that to be thread safe and isolated is a lot of work. LLVM is probably the compiler that could most easily get this done since it has been designed to be used as a library from the beginning.
A common problem in linking problems has to do with circular dependencies. Suppose you have a program that looks like this:
Here program calls into function one, which is in library A. That calls into function two, which is in library B. Finally that calls into function three, which is back in library A again.
Let's assume that we use the following linker line to build the final executable:
gcc -o program prog.o liba.a libb.a
Because linkers were originally designed in the 70s, they are optimized for minimal resource usage. In this particular case the linker will first process the object file and then library A. It will detect that function one is used so it will take that function's implementation and then throw the rest of library A away. It will then process library B, take function two in the final program and note that function three is also needed. Because library A was thrown away the linker can not find three and errors out. The fix to this is to specify A twice on the command line.
This is how everyone has been told things work and if you search the Internet you will find many pages explaining this and how to set it up linker command lines correctly.
But is this what actually happens?
Let's start with Visual Studio
Suppose you were to do this in Visual Studio. What do you think would happen? There are four different possiblities:
Linking fails with missing symbol three.
Linking succeeds and program works.
Either 1. or 2. happens, but there is not enough information to tell which.
Linking succeeds but the final executable does not run.
The correct answer is 2. Visual Studio's linker is smart, keeps all specified libraries open and uses them to resolve symbols. This means that you don't have to add any library on the command line twice.
Onwards to macOS
Here we have the same question as above but using macOS's default LLD linker. The choices are also the same as above.
The correct answer is also 2. LLD keeps symbols around just like Visual Studio.
What about Linux?
What happens if you do the same thing on Linux using the default GNU linker? The choices are again the same as above.
Most people would probably guess that the correct answer here is 1. But it's not. What actually happens is 3. That is, the linking can either succeed or fail depending on external circumstances.
The difference here is whether functions one and three are defined in the same source file (and thus end up in the same object file) or not. If they are in the same source file, then linking will succeed and if they are in separate files, then it fails. This would indicate that the internal implementation of GNU ld does not work at the symbol level but instead just copies object files out from the AR archive wholesale if any of their symbols are used.
What does this mean?
For example it means that if you build your targets with unity builds, their entire symbol resolution logic changes. This is probably quite rare but can be extremely confusing when it happens. You might also have a fully working build, which breaks if you move a function from one file to another. This is a thing that really should not happen but when it does things get very confusing.
The bigger issue here is that symbol resolution works differently on different platforms. Normally this should not be an issue because symbol names must be unique (or they must be weak symbols but let's not go there) or the behaviour is undefined. It does, however, place a big burden on cross platform projects and build systems because you need to have very complex logic in place if you wish to deduplicate linker flags. This is a fairly common occurrance even if you don't have circular dependencies. For example when building GStreamer with Meson some time ago the undeduplicated linker line contained hundreds duplicated library entries (it still does but not nearly as many).
The best possible solution would be if GNU ld started behaving the same way as VS linker and LLD. That way all major platforms would behave the same and things would get a lot simpler. In the mean time one should be able to simulate this with linker grouping flags:
Go through all linker arguments and split them to libraries that use link_whole and those that don't. Throw away any existing linker grouping.
Deduplicate and put the former at the beginning of the link line with the requisite link_full arguments.
Deduplicate all entries in the list of libraries that don't get linked fully.
Put the result of 3 on the command line in a single linker group.
This should work and would match fairly accurately what VS and LLD already do, so at least all cross platform projects should work out of the box already.
What about other platforms?
The code is here, feel free to try it out yourself.
In a previous post we talked about Finland's Linux powered slot machines. It was mentioned that there are about 20 000 of these machines in total. It turns out that managing and maintaining all those machines is a not as easy as it may first appear.
In the modern time of The Cloud, 20 thousand machines might not seem like much. Basic cloud management software such as Kubernetes scales to hundreds of thousands, even millions of machines without even breaking a sweat. Having "only" 20 thousand machines may seem like a small and simple thing that can be managed by one intern in their spare time. In reality things get difficult as there are many unique challenges to managing slot machines as opposed to regular servers.
The data center
Large scale computer fleets are housed in data centers. Slot machines are not. They are scattered across Finland in supermarkets and gas stations. This means that any management solution based on central control is useless. Another way of looking at this is that the data center housing the machines is around 337 thousand square kilometers in size. It is left as an exercise to the reader to calculate the average distance between two nearest machines assuming they are distributed evenly over the surface area.
Every machine is needed
The mantra of current data center design is that every machine must be expendable. That is, any computer may break down at any time, but the end user does not notice this because all operations are hidden behind a reliable layer. Workloads can be transferred from one machine to another either in the same rack, or possibly even to the other side of the world without anyone noticing.
Slot machines have the exact opposite requirements. Every machine must keep working all the time. If any machine breaks down, money is lost. Transferring the work load from a broken machine in the countryside to Frankfurt or Washington is not feasible, because it would require also moving the players to the new location. This is not very profitable, as atoms are much more expensive and slow to transfer between continents than electrons.
The reliability requirements are further increased by the distributed locations of the machines. It is not uncommon that in the sparsely populated areas the closest maintenance person may be more than 400 km away.
The Internet connection
Data centers nowadays have 10 Gb Ethernet connections or something even faster. In contrast it is the responsibility of the machine operator to provide a net connection to a slot machine. This means that the connections vary quite a lot. At the lowest end are locations that get poor quality 3G reception some of the time.
Remote management is also an issue. Some machines are housed in corporate networks behind ten different firewalls all administered by different IT provider organisations, some of which may be outsourced. Others are slightly less well protected but flakier. Being able to directly access any machine is the norm in data centers. Devices housed in random networks do not have this luxury.
The money problem
Slot machines deal with physical money. That makes them a prime target for criminals. The devices also have no physical security: you must be able to physically touch them to be able to play them. This is a challenging and unusual combination from a security point of view. Most companies would not leave their production servers outside for people to fiddle around with, but for these devices it is a mandatory requirement.
The beer attack
Many machines are located in bars. That means that they need to withstand the forces of angry intoxicated players. And, as we all know, drunk people are surprisingly inventive. A few years ago some people noticed that the machines have ventilation holes. They then noticed that pouring a pint of beer in those holes would cause a short circuit inside the machine causing all the coins to be spit out.
This issue was fixed fairly quickly, because you really don't want to be in a situation where drunk people would have financial motivation to pour liquids on high voltage equipment in crowded rooms. This is not a problem one has to face in most data centers.
Update challenges
There are roughly two different ways of updating an operating system install: image based updates and package based updates. Neither of these works particularly well in slot machine usage. Games are big, so downloading full images is not feasible, especially for machines that have poor network connections. Package based updates have the major downside that they are not atomic. In desktop and server usage this is not really an issue because you can apply updates at a known good time. For remote devices this does not work because they can be powered off at any time without any warning. If this happens during an upgrade you have a broken machine requiring a physical visit from a maintenance person. As mentioned above this is slow and expensive.
