On the Meson manual crowdfunding page it is mentioned that the end result can not be put under a fully free license. Several people have said that they "don't believe such a law could exist" or words to that effect. This blog post is an attempt to to explain the issue in English as all available text about the case is in Finnish. As a disclaimer: I'm not a lawyer, the following is not legal advice, there is no guarantee, even that any of the information below is factual.
To get started we need to go back in time a fair bit and look at disaster relief funds. In Finland you must obtain a permit from the police in order to gather money for general charitable causes. This permit has strict requirements. The idea is that you can't just start a fundraising, take people's money and pocket it, instead the money must provably go to the cause it was raised for. The way the law is written is that a donation to charity is done without getting "something tangible" in return. Roughly if you give someone money and get a physical item in return, it is considered a sales transaction. If you give money to someone and in return get a general feeling of making the world better in some way, that is considered a donation. The former is governed by laws of commerce, the latter by laws of charity fundraising.
A few years ago there was a project to create a book to teach people Swedish. The project is page is here, but it is all in Finnish so it's probably not useful to most readers. They had a crowdfunding project to finish the project with all the usual perks. One of the goals of the crowdfunding was to make the book freely distributable after publishing. This is not unlike funding feature work on FOSS projects works.
What happened next is that the police stepped in and declared this illegal (news story, in Finnish). Their interpretation was that participating in this campaign without getting something tangible in return (i.e. paying less than the amount needed to get the book) was a "charitable donation". Thus it needs a charity permit as explained above. Running a crowdfunding campaign is still legal if it is strictly about pre-sales. That is, every person buys "something" and that something needs to have "independent value" of some sort. If the outcome of a project is a PDF and that PDF becomes freely available, it can be argued that people who participated did not get any "tangible value" in exchange for their money.
Because of this the outcome of the Meson manual crowdfunding campaign can not be made freely available. This may seem a bit stupid, but sadly that's the law. The law is undergoing changes (see here, in Finnish), but those changes will not take effect for quite some time and even when they do it is unclear how those changes would affect these kinds of projects.
The Meson Build system has existed for over six years. Even though we have a fairly good set of documentation, there has not been a standalone user's manual for it. Until now.
A crowdfunding campaign to finance the manual has just been launched on Indiegogo. The basic deal is simple, for 30€ you get the final book as a PDF. To minimize work and save trees, there is no physical version. There are also no stickers, beer mats or any other tchotchkies. There are a few purchase options as well as opportunities for corporate sponsorships. Please see the Indiegogo project page for further details. If there are any questions about this campaign feel free to contact me. The easiest way is via email.
Overall I'm quite excited about this campaign. One reason is obviously personal, but the other has to do with sustainability of FOSS projects in general. There has been a lot of talk about how maintainers of open source projects can get compensated for their work. This campaign can be seen as an experiment to see if the crowdfunding model could work in practice.
So if you are just getting started with building software and want a user manual, buy this book. If you have basic experience with Meson and want to dive deeper, buy this book. If you are a seasoned veteran and don't really need a book but want to support the project (specifically me), buy this book. Regardless of anything else, please spread the word on your favourite social media and real world venues of choice.
Do not, I repeat, NOT make your test framework fail a test run if it writes any text to stderr! No matter how good of on idea you think it is, it's terrible.
If you absolutely, positively have to do that, then print the reason for this failure in your output log. If you can't think of a proper warning message, feel free to copy paste this one:
THIS TEST FAILED BECAUSE IT WROTE TO STDERR AND SOMEONE HERE (OBVIOUSLY NOT ME) THOUGHT MAKING THAT A HARD ERROR WOULD BE A GOOD IDEA!!!!!!!
Sincerely: a person who has lost hours of his life on this sh*t on multiple occasions and can never get it back.
It is fairly well established that Linux is not the #1 game development platform at this point in time. It is also unlikely to be the most popular one any time soon due to reasons of economics and inertia. A more interesting question, then, would be can it be made the best one? The one developers favour over all others? The one that is so smooth and where things work so well together that developers feel comfortable and productive in it? So productive that whenever they use some other environment, they get aggravated by being forced to use such a substandard platform?
Maybe.
The requirements
In this context we use "game development" as a catch all term for software development that has the following properties:
The code base is huge, millions or tens of millions lines of code.
Non-code assets are up to tens of gigabytes in size
A program, once built, needs to be tested on a bunch of various devices (for games, using different 3D graphics cards, amounts of memory, processors etc).
What we already have
A combination of regular Linux userland and Flatpak already provides a lot. Perhaps the most unique feature is that you can get the full source code of everything in the system all the way down to the graphic cards' device drivers (certain proprietary hardware vendors notwithstanding). There is no need to guess what is happening inside the graphics stack, you can just add breakpoints and step inside it with full debug info.
Linux as a platform is also faster than competing game development systems at most things. Process invocation is faster, file systems are faster, compilations are faster, installing dependencies is faster. These are the sorts of small gains that translate to better productivity and developer happiness.
Flatpak as a deployment mechanism is also really nice.
What needs to be improved?
Many tools in the Linux development chain assume (directly or indirectly) that doing something from scratch for every change is "good enough". Once you get into large enough scale this no longer works. As an example flatpak-builder builds its packages by copying the entire source tree inside the build container. If your repository is in the gigabyte range this does not work, but instead something like bind mounting should be used. (AFAIK GNOME Builder does something like this already.) Basically every operation needs to be O(delta) instead of O(whole_shebang):
Any built program that runs on the developer's machine must be immediately deployable on test machines without needing to do a rebuild on a centralised build server.
Code rebuilds must be minimal.
Installs must skip files that have not changed since the last install.
Package creation must only account for changed files.
All file transfers must be delta based. Flatpak already does this for package downloads but building the repo seems to take a while.
A simple rule of thumb for this is that changing one texture in a game and deploying the result on a remote machine should not take more than 2x the amount of time it would take to transfer the file directly over with scp.
Other tooling support
Obviously there needs to be native support for distributed builds. Either distcc, IceCream or something fancier, but even more important is great debugging support.
By default the system should store full debug info and corresponding source code. It should also log all core dumps. Pressing one button should then open up the core file in an IDE with up to date source code available and ready to debug. This same functionality should also be usable for debugging crashes in the field. No crash should go unstored (assuming that there are no privacy issues at play).
Perhaps the hardest part is the tooling for non-coders. It should be possible to create new game bundles with new assets without needing to touch any "dev" tools, even when running a proprietary OS. For example there could be a web service where you could do things like "create new game install on machine X and change model file Y with this uploaded file Z". Preferably this should be doable directly from the graphics application via some sort of a plugin.
Does something like this already exist?
Other platforms have some of these things built in and some can be added with third party products. There are probably various implementations of these ideas inside the closed doors of many current game development studios. AFAICT there does not exist a fully open product that would combine all of these in a single integrated whole. Creating that would take a fair bit of work, but once done we could say that the simplest way to set up the infrastructure to run a game studio is to get a bunch of hardware, open a terminal and type:
In interesting intellectual design challenge is to take a working thing (library, architecture, etc) and then see what would happen if you would reimplement it with the exact opposite way. Not because you'd use the end result anywhere, but just to see if you can learn something new. Or, in other words:
As an example let's apply this approach to C++'s fixed size array object or std::array. Some of its core design principles include:
Indexing is by default unsafe, user is responsible for providing valid indexes.
Errors are reported via exceptions.
Iterators may be invalid and invoking invalid iterators is allowed but UB.
Indexing must be exactly as fast as accessing a C array.
Inverting all of these give us the following design principles:
Out of bound accesses must be impossible.
No exceptions (assuming contained objects do not throw exceptions).
All iterator dereferences are guarded and may never lead to bad accesses.
It's ok to use some (but not much) processor time to ensure safety. Aim for zero overhead when possible.
So how does it look like?
An experimental PoC implementation can be found in this Github repo. Note that the code is intentionally unpolished. There are silly choices made. That is totally fine, it's not meant for actual use, only to explore the problem space.
The most important operation for an array type is indexing. It must work for all index values, even for those out of bounds. As no exceptions are allowed, the natural way to make this work is to return a Maybe type. This could be a std::optional<std::reference_wrapper<T>>, but for reasons that will become apparent later, we use a custom type. The outcome for this is kind of nice, allowing you to do things like (assuming an array of size 20):
assert(x[0]); // check if index is valid.
assert(!x[20]); // OOB indexes are invalid
*x[0] = 5 // assignment
assert(*x[0] == 5); // dereference
int i = *x[20]; // aborts program
The overall usability is kind of nice. This is similar to languages that have int? variables. The biggest problem here is that there is no way to prevent dereferencing an invalid maybe object, leading to termination. A typical bug would look like this:
if(maybe) {
*maybe = 3;
} else {
*maybe = 4; // Legal code but should not be.
}
There are at least three possible architectural ways to solve this:
Pattern matching (a switch/case on the object type) with language enforcement.
A language construct for "if attempting to use an invalid maybe object, exit the current function with an error". There have been talks of a try statement that would do something like this.
Maybes can not be dereferenced, only called with a method like visit(if_valid_function, if_not_valid_function).
#3 can be done today but is tedious and still does not permit automatically returning an error from the current function block.
Iteration
Creating a safe iterator is fairly simple. This iterator has a pointer to the original object and an integer offset. Dereferencing it calls the indexing operation and returns the maybe to the caller. This works fine until you test it with std::sort and after a lot of debugging find out that the implementation has a code block looking like this:
T tmp = std::move(a);
a = std::move(b);
b = std::move(tmp);
Even if you have swap defined for your objects, it will call this at some point. The problem here is that since the temporary does not point to any existing object, it can not store the value of the first move. There does not seem to be a good solution for this. Swap-only types might work, but such a type can not be defined with C++'s type system (or at least std::sort can not handle such types). The solution used in this example is that if a maybe object is created so that it does not point to any existing object, it stores objects inside itself. This works (for this use case) but feels a bit iffy.
Another problem this exposes is that the STL does not really work with these kinds of types. The comparison function returns a boolean, but if you compare (for whatever reason) invalid iterators and don't use exceptions there should be three different return values: true, false and erroneous_comparison. That is, an std::optional<bool>. Fixing this properly would mean changing std::sort to handle failing comparisons.
But what about performance?
For something like sorting, checking every access might cause a noticeable performance hit. If you do something like this:
std::sort(coll.begin(), coll.end())
it is fairly easy to verify that the range is valid and all thus accesses will be valid (assuming no bugs in the sort implementation). It would be nice to be able to opt out of range checks in these cases. Here's one way of doing it:
Here unsafe is a helper class that returns raw pointers to the underlying object. In this way individual accesses (which are bug prone) are guarded but larger batch operations (which are safer) can optionally request a faster code path. The unsafe request is immediately apparent and easily greppable.
Is this fully safe?
No. There are at least three different ways of getting an invalid reference.
Deleting the main object while iterators/maybes are outstanding. Would require tracking all outstanding objects. Probably too heavyweight. In Rust this is handled by the borrow checker at compile time.
Unallocating the memory backing the object without running its destructor. This is actually legal. Can't be guarded against.
Changing the object's backing memory to read only with a syscall. Can't be guarded against.
Most FOSS projects have a bug tracker, code repository and an IRC channel. Many even have a developer meeting, conference or awards given out to most prolific contributors. But do you know what no FOSS project has that they really should?
A Dragon Ball/NES/SNES style RPG quest.
To fill this void in the FOSS experience we have taken it upon ourselves to create one. The first thing any quest needs is artifacts worth fighting over. Going with the style of Dragon Ball we have created six power glasses infused with the power of software building. This is what they look like.
These items were hand-crafted in a blazing Finnish furnace in the dead of winter. Once cast and cooled the Meson logo and the developer team's secret motto were debossed on the side by artisanal free range engraving elves of Espoo.
To make the quest more challenging, we have scattered all the power glasses to the extreme ends of the world. Each is being held by a mini boss who must be defeated in order to obtain the corresponding power. In no particular order they are:
Igor Gnatenko, Czech republic, holding the power of the First Community Member
Nirbheek Chauhan, India, holding the power of the Co-Maintainer
Dylan Baker, USA, holding the power of 3D Graphics
Alexis Jeandet, France, holding the power of CI
Aleksey Filippov, Ireland, holding the power of Dependency Packaging
They have all stood as guardians of their respective power for an extended amount of time and for that they deserve vast heaps of praise and gratitude from all of us.
The sixth and last power glass is held under strict supervision in my Finnish volcano lair headquarters [1] and can only be unlocked when holding all the other five power glasses.
It is currently unclear whether this system behaves in the Mega Man style where you can use the powers of one boss against another or in the flat style where you get a fixed amount of XP and the battle order does not matter.
The only thing remaining is the final question, namely what will happen once all six power glasses have been reunited? We have not actually thought that far ahead yet. Considering that all of this is stolen from Dragon Ball, probably something fairly anticlimactic.
Post scriptum
Please don't actually go out and fight these people and steal their glassware. It would be very inconvenient for everyone involved.
[1] Not guaranteed to contain an actual volcano, or even a replica of same.
My previous blog post on the future of Meson was a smashing hit (with up to five(!) replies on Twitter), so I figured I'd write a bit more in depth article about the issue. I'd like to emphasize that like previously, this article only mentions Rust by name, but the issues raised are identical for all new programming languages such as D, Go, Nim, and all the others whose names I don't even know.
Let's start with a digression.
Never use the phrase "why don't you just"!
Whenever you discuss on the Internet and some sort of major problem appears, someone will come up and say something along the lines of "well, why don't you just do X", usually as the very first response.
Please don't do that.
Using this phrase (or any variation thereof) makes the person saying seem to do one of two things:
Trying to handwave the actual problem away by belittling it.
Not understanding (or even wanting to understand) the problem and instead blindly pushing their own agenda.
Note that even if you are not doing this, other people will assume that you are. It does not really matter what you think or know or say, only how other people perceive your message. This is perfectly demonstrated in this clip from the movie Sneakers.
Using phrases like "why don't you just" make you look bad, even if your message and intentions were good.
Well why don't you just call Cargo then?
Why indeed. This is the most common reply I get when this issue arises. So let's dive deep to find out from first principles why this is. Usually when people say this they envision a project that looks something like this.
That is, there is some standalone piece of Rust code and then Meson adds something on top (or builds a completely different executable or something). The two are fairly well separated and doing their own things. For these kinds of projects having one call the other works fine. If this was the case we could all go home. But let's make the project slightly more complex.
This this case we have an executable and a library liba that are built with Meson. In addition there is a Rust library libb that also uses liba internally. This is where things get complicated.
The first thing to note is that in this case liba must onlybe built once and both the Rust library and the executable must use the same version of the library. Any solution that would require having two versions of liba are not acceptable. This means that there are two different interfaces between these build systems. The first is the injection interface by which Meson tells Cargo where and how it built liba. The second is the extraction interface by which Meson gets information from Cargo about what it built and how. Neither of these exist in a form that would be actually useful.
For extraction Cargo has an unstable feature called build-plan, which spits out the compilation commands Cargo would take. The idea is that the outer build system could take these commands, run them on their own and then things would work. But they don't because compilation arguments do not have all the information necessary. Suppose both these libraries are linked statically. Then linking against libb is not enough and build will fail, you also need to link against liba (and in a very specific order even). If the libraries are shared, then you should not add liba on the link line. The extraction interface would need to have a full serialisation of these dependencies. Otherwise the end result is not guaranteed to work and depends on random things such as whether the internal use of liba is before or after libb on the executable's link command line. Consuming of libraries does not work without this.
The injection interface is even trickier. Based on a quick review, many Rust libraries either build their deps themselves (which is bad) or call pkg-config (which is good if your deps come from the system, but in this case they don't). But to make this use case work there would be a need to tell from "the outside" how to obtain which library and what their dependencies are. As far as I know there is no way to tell this to Cargo and any project that does its own thing with build.rs is automatically guaranteed to not work without code changes.
And that was the easy part
Suppose that in one of the libraries generates headers during compilation. An example would be Protocol Buffers. You can't start compiling any code that depends on those headers until they exist. You could try to work around this by building one project at a time. This is the model of recursive Make and Visual Studio solution files. Its main downside is that it makes builds a lot slower. The interface specification would need to have the header information to get reliable builds.
Another thing about Protocol Buffers is that you need to build the compiler executable in order to compile proto definitions to sources. Like with headers there needs to be a way to tell the project to use the recently built protoc rather than trying to find it on its own. This also adds a new dependency between the projects. The interface specification needs to have all this information to get reliable builds.
All of this also needs to be in the extraction interface, too, in case library b generated headers or has an executable that should be used later on in the build.
The interface seems way too complicated...
Going through all the requirements you eventually find that in order to make things really, actually, work you need to serialise pretty much everything between the two projects. It would also need to be fully programming language agnostic. If we started the design work today it would take months or even years to get something working. Fortunately we don't necessarily have to.
This is exactly what Meson is.
All the work we have put in Meson for the past six years has been done to create a programming language independent way of describing builds and making them work together. As a side effect we have also created a reference implementation for such a build system that can be used to build code. But that is mostly incidental, the actual important thing is, and always has been, the description language. Every problem mentioned in this post has a solution in the Meson language and they have been used in production for year. Of course there would be problems and bugs found as the usage would grow but it's the same with every program. Bugs can be fixed.
What would it look like?
The build definition for library b that would support all the things discussed above would look like this (assuming the library has a pre-existing .h file and does not have a build.rs file):
project('libb', 'rust')
a_dep = dependency('a', fallback: ['a', 'a_dep'])
b = static_library('b', 'b.rs', dependencies: a_dep)
And that's it. This is all you need to get full project interop (there may be some implementation bugs).
Best of all this can be implemented in 100% Rust without adversely affecting existing projects. Either Cargo can parse the toml definitions and export this file, or Cargo could be made to read Meson build definitions natively. As an extra bonus any Rust project that uses external dependencies could use this to get native and transparent pkgconfig support without needing to write a build.rs (obviously it would not work with Cargo proper unless support is added to Cargo to read Meson files).
Extra language advantage
There are many languages that want to become the next big systems programming language. Almost all of them advertise good C compatibility as a big selling point. There is not a single one, though, that has "good interoperability with C tooling" in their priority list. In fact it is often quite the opposite. Getting language specific build systems working within existing projects can be a massive pain because they have not been designed for that.
Thus if a language would take tooling compatibility as a priority and would produce Meson files, either as their build system's native format or through some sort of an export mechanism, it would have an ease of integration advantage over other languages. Libraries written in language X could be dropped inside other projects that build with Meson and they would basically work out of the box (again, modulo bugs).
So is this the One True Way of how things will go?
No. Of course not. There are many different ways to solve this problem and this is only one possible way (in my opinion a fairly good one, but obviously other people will have different opinions and priorities). The real solution will, at least in the short term, probably be something completely different. But in any case it should fulfil the requirements discussed above because you can't really combine two languages and get a usable and reliable combination without them.
