torstai 14. helmikuuta 2019

Introducing the Meson Quest

Most FOSS projects have a bug tracker, code repository and an IRC channel. Many even have a developer meeting, conference or awards given out to most prolific contributors. But do you know what no FOSS project has that they really should?

A Dragon Ball/NES/SNES style RPG quest.

To fill this void in the FOSS experience we have taken it upon ourselves to create one. The first thing any quest needs is artifacts worth fighting over. Going with the style of Dragon Ball we have created six power glasses infused with the power of software building. This is what they look like.


These items were hand-crafted in a blazing Finnish furnace in the dead of winter. Once cast and cooled the Meson logo and the developer team's secret motto were debossed on the side by artisanal free range engraving elves of Espoo.

To make the quest more challenging, we have scattered all the power glasses to the extreme ends of the world. Each is being held by a mini boss who must be defeated in order to obtain the corresponding power. In no particular order they are:

  • Igor Gnatenko, Czech republic, holding the power of the First Community Member
  • Nirbheek Chauhan, India, holding the power of the Co-Maintainer
  • Dylan Baker, USA, holding the power of 3D Graphics
  • Alexis Jeandet, France, holding the power of CI
  • Aleksey Filippov, Ireland, holding the power of Dependency Packaging
They have all stood as guardians of their respective power for an extended amount of time and for that they deserve vast heaps of praise and gratitude from all of us.

The sixth and last power glass is held under strict supervision in my Finnish volcano lair headquarters [1] and can only be unlocked when holding all the other five power glasses.

It is currently unclear whether this system behaves in the Mega Man style where you can use the powers of one boss against another or in the flat style where you get a fixed amount of XP and the battle order does not matter.

The only thing remaining is the final question, namely what will happen once all six power glasses have been reunited? We have not actually thought that far ahead yet. Considering that all of this is stolen from Dragon Ball, probably something fairly anticlimactic.

Post scriptum

Please don't actually go out and fight these people and steal their glassware. It would be very inconvenient for everyone involved. 

[1] Not guaranteed to contain an actual volcano, or even a replica of same.

sunnuntai 10. helmikuuta 2019

Why is cross-language cooperation so hard?

My previous blog post on the future of Meson was a smashing hit (with up to five(!) replies on Twitter), so I figured I'd write a bit more in depth article about the issue. I'd like to emphasize that like previously, this article only mentions Rust by name, but the issues raised are identical for all new programming languages such as D, Go, Nim, and all the others whose names I don't even know.

Let's start with a digression.

Never use the phrase "why don't you just"!

Whenever you discuss on the Internet and some sort of major problem appears, someone will come up and say something along the lines of "well, why don't you just do X", usually as the very first response.

Please don't do that.

Using this phrase (or any variation thereof) makes the person saying seem to do one of two things:
  1. Trying to handwave the actual problem away by belittling it.
  2. Not understanding (or even wanting to understand) the problem and instead blindly pushing their own agenda.
Note that even if you are not doing this, other people will assume that you are. It does not really matter what you think or know or say, only how other people perceive your message. This is perfectly demonstrated in this clip from the movie Sneakers.



Using phrases like "why don't you just" make you look bad, even if your message and intentions were good.

Well why don't you just call Cargo then?

Why indeed. This is the most common reply I get when this issue arises. So let's dive deep to find out from first principles why this is. Usually when people say this they envision a project that looks something like this.


That is, there is some standalone piece of Rust code and then Meson adds something on top (or builds a completely different executable or something). The two are fairly well separated and doing their own things. For these kinds of projects having one call the other works fine. If this was the case we could all go home. But let's make the project slightly more complex.


This this case we have an executable and a library liba that are built with Meson. In addition there is a Rust library libb that also uses liba internally. This is where things get complicated.

The first thing to note is that in this case liba must only be built once and both the Rust library and the executable must use the same version of the library. Any solution that would require having two versions of liba are not acceptable. This means that there are two different interfaces between these build systems. The first is the injection interface by which Meson tells Cargo where and how it built liba. The second is the extraction interface by which Meson gets information from Cargo about what it built and how. Neither of these exist in a form that would be actually useful.

For extraction Cargo has an unstable feature called build-plan, which spits out the compilation commands Cargo would take. The idea is that the outer build system could take these commands, run them on their own and then things would work. But they don't because compilation arguments do not have all the information necessary. Suppose both these libraries are linked statically. Then linking against libb is not enough and build will fail, you also need to link against liba (and in a very specific order even). If the libraries are shared, then you should not add liba on the link line. The extraction interface would need to have a full serialisation of these dependencies. Otherwise the end result is not guaranteed to work and depends on random things such as whether the internal use of liba is before or after libb on the executable's link command line. Consuming of libraries does not work without this.

The injection interface is even trickier. Based on a quick review, many Rust libraries either build their deps themselves (which is bad) or call pkg-config (which is good if your deps come from the system, but in this case they don't). But to make this use case work there would be a need to tell from "the outside" how to obtain which library and what their dependencies are. As far as I know there is no way to tell this to Cargo and any project that does its own thing with build.rs is automatically guaranteed to not work without code changes.

And that was the easy part

Suppose that in one of the libraries generates headers during compilation. An example would be Protocol Buffers. You can't start compiling any code that depends on those headers until they exist.  You could try to work around this by building one project at a time. This is the model of recursive Make and Visual Studio solution files. Its main downside is that it makes builds a lot slower. The interface specification would need to have the header information to get reliable builds.

Another thing about Protocol Buffers is that you need to build the compiler executable in order to compile proto definitions to sources. Like with headers there needs to be a way to tell the project to use the recently built protoc rather than trying to find it on its own. This also adds a new dependency between the projects. The interface specification needs to have all this information to get reliable builds.

All of this also needs to be in the extraction interface, too, in case library b generated headers or has an executable that should be used later on in the build.

The interface seems way too complicated...

Going through all the requirements you eventually find that in order to make things really, actually, work you need to serialise pretty much everything between the two projects. It would also need to be fully programming language agnostic. If we started the design work today it would take months or even years to get something working. Fortunately we don't necessarily have to.

This is exactly what Meson is.

All the work we have put in Meson for the past six years has been done to create a programming language independent way of describing builds and making them work together. As a side effect we have also created a reference implementation for such a build system that can be used to build code. But that is mostly incidental, the actual important thing is, and always has been, the description language. Every problem mentioned in this post has a solution in the Meson language and they have been used in production for year. Of course there would be problems and bugs found as the usage would grow but it's the same with every program. Bugs can be fixed.

What would it look like?

The build definition for library b that would support all the things discussed above would look like this (assuming the library has a pre-existing .h file and does not have a build.rs file):

project('libb', 'rust')

a_dep = dependency('a', fallback: ['a', 'a_dep'])
b = static_library('b', 'b.rs', dependencies: a_dep)
b_dep = declare_dependency(link_with: b, include_directories: '.')

And that's it. This is all you need to get full project interop (there may be some implementation bugs).

Best of all this can be implemented in 100% Rust without adversely affecting existing projects. Either Cargo can parse the toml definitions and export this file, or Cargo could be made to read Meson build definitions natively. As an extra bonus any Rust project that uses external dependencies could use this to get native and transparent pkgconfig support without needing to write a build.rs (obviously it would not work with Cargo proper unless support is added to Cargo to read Meson files).

Extra language advantage

There are many languages that want to become the next big systems programming language. Almost all of them advertise good C compatibility as a big selling point. There is not a single one, though, that has "good interoperability with C tooling" in their priority list. In fact it is often quite the opposite. Getting language specific build systems working within existing projects can be a massive pain because they have not been designed for that.

Thus if a language would take tooling compatibility as a priority and would produce Meson files, either as their build system's native format or through some sort of an export mechanism, it would have an ease of integration advantage over other languages. Libraries written in language X could be dropped inside other projects that build with Meson and they would basically work out of the box (again, modulo bugs).

So is this the One True Way of how things will go?

No. Of course not. There are many different ways to solve this problem and this is only one possible way (in my opinion a fairly good one, but obviously other people will have different opinions and priorities). The real solution will, at least in the short term, probably be something completely different. But in any case it should fulfil the requirements discussed above because you can't really combine two languages and get a usable and reliable combination without them.

tiistai 5. helmikuuta 2019

On possible futures of Meson

At FOSDEM I talked to a bunch of people about an issue that has been brought up a couple of times recently, specifically that of integrating Rust in existing code bases. Some projects are looking into converting parts (or presumably eventually everything) of their code to Rust. A requirement of this is that for some time there need to be both Rust and C or whatever language within one project at the same time. (The rest of this blog post will use Rust as an example, but the same issues are present in all modern programming languages that have the same build system/dependency setup. In practice this means almost all of them.)

This would not be such a problem except that Rust by itself has pretty much nothing in the standard library and you need to get many crates via Cargo for even fairly simple programs. Several people do not seem particularly thrilled about this for obvious reasons, but have given up on this battle because "in practice it's impossible to develop in Rust without using Cargo" or words to that effect. As the maintainer of Meson, they obviously come to me with the integration problem. Meson does support compiling Rust directly, but it does not go through Cargo.

This is where I'm told to "just call Cargo" instead. There are two major problems with this. The first one is technical and has to do with the fact that having two different build systems and dependency managers in one build directory does not really work. We're not going to talk about this issue in this blog post, interested people can find writings about this issue using their favorite bingoogle. The second issue is non-technical, and the more serious one.

This approach means the death of Meson as a project

If we assume (as seems to be the case currently) that new programming languages such as Rust take over existing languages and that they all have their own build system, then Meson becomes unnecessary. Having a build system whose only task is to call a different build system is useless. It is nothing but bloat that should be deleted. As the person whose (unpaid) job it is to care about the long term viability of Meson, this does not make me feel particularly good.

So what might the future might in hold? Let's look at some alternatives.

New languages take over and keep their own build systems

New things take over, old code gets rewritten or thrown away. Each language keeps living in its own silo. Language X is better than language Y battles become the new Emacs vs vi battles. Meson becomes obsolete and gets shipped to the farm where it joins LISP machines, BeOS and Amigas as technology that was once interesting and nice but then was forgotten.

This is where things are currently heading. It is also the most likely outcome.

New languages adopt Meson as their build system

Rather than building and maintaining their own build systems, new languages switch to using Meson so everyone has a shared system and things work together. This will not happen. The previous statement is so important that I'll write it here for a second time, but this time in bold and on its own line:

This will not happen!

Every new language wants to provide a full toolchain and developer experience on their own. This is actually a good choice because it means that they can provide a full dev experience by themselves without depending on any external tooling (especially one that is written in a different programming language). Being able to e.g. just install one MSI package on Windows and have the full dev experience there is a really nice thing to have. Because of this no new programming language will accept an external build tool. No way. No how.

Meson becomes a glue layer to combine programming languages

A common suggested solution is that language native build systems export their state somehow so that external tools can drive them. Meson would then take these individual projects and make them work together. On the face of it this seems simple and workable and allows mixing languages freely. But if you think about it more, this is not a great place to be in. Basically it means that your project's purpose changes from being a build system to being responsible of all project integration but who does not have any say on how builds happen. There is also no leverage on the actual toolchain developers.

A condensed description of this task would be integration hell as a service. It is unlikely that anyone would want to spend their free time doing that. I sure as hell don't. I deal with enough of that stuff at work already.

So is everything terrible and all hope lost forever?

Maybe.

But then again maybe not.

Meson the language is not Meson the implementation

Meson was designed from the beginning to be programming language independent. This means both the languages it supports and the language it is implemented in. It does not leak the fact that it's implemented in Python to the build definition language. The core Meson functionality is only a few thousand lines of Python, reimplementing it is not a massive task. There is nothing wrong with having multiple implementations. In fact it is encouraged. Feel like rewriting it in Rust? Go for it (pun not intended).

In this approach new programming languages would get to keep their own toolchains, installs and build systems. The only thing that would change is the build description language. The transition would take a fair bit of work to be sure, but the benefit is that all that code becomes easily integratable with other languages. The reference implementation of language X could even choose to only support its own language and hand multi-language projects off to the reference implementation.

There are other advantages as well. For example currently (or the last time I looked, which admittably was some time ago) Cargo farms almost all non-standard work to build.rs, leading to wheel reinvention. Meson has native support for most of the things a project might need (source code generation etc). Thus each language would not need to design their own DSL for describing these, but could instead take Meson's and implement it in their own tools. Meson also has a lot of functionality for things like creating Python modules, which you would also get for free.

But the main advantage of all of this would be cooperation. You could mix and match languages with ease (assuming a working API/ABI, but that is outside the build system's purview) and switch between different languages without needing to learn yet another build definition language and toolchain behaviour.

The main downside of this approach is that it can not be injected from "the outside". It would only work if there are people inside the language communities that would consider this a worthwhile thing to do and would drive the change from within. Even for them it would probably be a long battle, because these sorts of changes have traditionally met with a lot of resistance.

How will all this affect Meson development?

In the short term probably not in any noticeable way.

And in the long term?


lauantai 2. helmikuuta 2019

Meson's new logo: a design process

At FOSDEM, quite literally only a few moments ago, we published the new Meson logo.



The design process was fairly long and somewhat arduous. This blog post aims to provide the rationale for all aspects of the design process.

The shape

From the very beginning it was decided that the logo's shape should be based on the Reuleaux triangle. The overall shape is smooth and pleasant, yet mathematically precise. In spite of its simplicity it can be used for surprisingly complex things. Perhaps the best known example is that if you have a drill bit shaped like a Reuleaux triangle, it can be used to create a rectangular hole. Those who have knowledge about compiler toolchains know that these sort of gymnastics are exactly what a build system needs to do. 

There were tens of different designs that tried to turn this basic idea into a logo. In practice it meant sitting in front of Inkscape and trying different things. These attempts included such things as combining the Reuleaux triangle with other shapes such as circles and triangles, as well as various attempts to build a stylished "M" letter. None of them really clicked until one day I put a smaller triangle upside down inside the other. Something about that shape was immediately captivating.



This shape was not directly usable by itself because it looked a lot like the Sierpinski triangle, which you may know from many corporate logos and other media. People also felt that this shape had an eye or a face that was looking at them, which was distracting. The final breakthrough came when even smaller versions of the triangle were added to the corners of the smaller triangle to produce a roughly propeller-like shape. 

The colour

Meson's original logo was green as it is the colour of growth and nature. This is a common choice for tooling. As an example, Bazel's original logo was a stylised green leaf. It is easy to see why green does not work with the chosen shape:



Legend of Zelda alliterations aside, some other color was needed. I was reading an article about logo colors which mentioned that for some reason purple logos are not as common as one would imagine. Upon reading that sentence the choice was done. After all if it is good enough for Taco Bell and Barbie, it is good enough for us.

Choosing the actual shade of purple was a tougher problem. We wanted a deep shade that is closer to blue than red. A bunch of back and forth got us to the current selection, PANTONE 2105C. This decision might not be final, though. The shade is slightly off from what I considered the perfect one. It would also be nice to have a free-as-in-freedom way of defining the logo colour rather than using one from a proprietary vendor.

The font

Technical logos usually use sans serif fonts, possibly with a geometrical shape to evoke stability and rigidity. This was the original plan for Meson's logo as well but eventually it seemed like those fonts were fairly heavy. We wanted the text to be light to symbolize the lean internals of Meson and its low resource usage. It also turned out that a light text nicely balances the slightly bulky logo shape. Going through various font and design sites I eventually found myself looking at this picture.


This is the inscription in Trajan's Column as created by ancient Romans. The typeface was exactly as needed, light but a definite sense of gravitas. The basic letterforms were traced and after several rounds of artisanal hand-crafted vector object tuning the final text shape was finished. For comparison we also tested Computer Modern as the logo text. It got a very sharp 50/50 split of people either hating it or loving it.

Licenses

The logo is not licensed under Apache as the rest of Meson is. It turns out that there is a license hole in the FOSS world for logos. It would be great if someone could come up with a simple and clear cut "you can use the logo to refer to the project but not claim endorsement of the project or use the logo on any merchandising" license. Many projects, such as GNOME, have their own license for this but copying random license legalese always feels a bit scary.

Final words

If you spend long enough working on any abstract geometric shape, you are going to think of some real world object that resembles it. That's just how the human brain works. This the image my subconscious brought forth.


I swear that this was not an intentional design point.