Why is cross-language cooperation so hard?

My previous blog post on the future of Meson was a smashing hit (with up to five(!) replies on Twitter), so I figured I'd write a bit more in depth article about the issue. I'd like to emphasize that like previously, this article only mentions Rust by name, but the issues raised are identical for all new programming languages such as D, Go, Nim, and all the others whose names I don't even know.

Let's start with a digression.

Never use the phrase "why don't you just"!

Whenever you discuss on the Internet and some sort of major problem appears, someone will come up and say something along the lines of "well, why don't you just do X", usually as the very first response.

Please don't do that.

Using this phrase (or any variation thereof) makes the person saying seem to do one of two things:

1. Trying to handwave the actual problem away by belittling it.
2. Not understanding (or even wanting to understand) the problem and instead blindly pushing their own agenda.

Note that even if you are not doing this, other people will assume that you are. It does not really matter what you think or know or say, only how other people perceive your message. This is perfectly demonstrated in this clip from the movie Sneakers.

Using phrases like "why don't you just" make you look bad, even if your message and intentions were good.

Well why don't you just call Cargo then?

Why indeed. This is the most common reply I get when this issue arises. So let's dive deep to find out from first principles why this is. Usually when people say this they envision a project that looks something like this.

That is, there is some standalone piece of Rust code and then Meson adds something on top (or builds a completely different executable or something). The two are fairly well separated and doing their own things. For these kinds of projects having one call the other works fine. If this was the case we could all go home. But let's make the project slightly more complex.

This this case we have an executable and a library liba that are built with Meson. In addition there is a Rust library libb that also uses liba internally. This is where things get complicated.

The first thing to note is that in this case liba must only be built once and both the Rust library and the executable must use the same version of the library. Any solution that would require having two versions of liba are not acceptable. This means that there are two different interfaces between these build systems. The first is the injection interface by which Meson tells Cargo where and how it built liba. The second is the extraction interface by which Meson gets information from Cargo about what it built and how. Neither of these exist in a form that would be actually useful.

For extraction Cargo has an unstable feature called build-plan, which spits out the compilation commands Cargo would take. The idea is that the outer build system could take these commands, run them on their own and then things would work. But they don't because compilation arguments do not have all the information necessary. Suppose both these libraries are linked statically. Then linking against libb is not enough and build will fail, you also need to link against liba (and in a very specific order even). If the libraries are shared, then you should not add liba on the link line. The extraction interface would need to have a full serialisation of these dependencies. Otherwise the end result is not guaranteed to work and depends on random things such as whether the internal use of liba is before or after libb on the executable's link command line. Consuming of libraries does not work without this.

The injection interface is even trickier. Based on a quick review, many Rust libraries either build their deps themselves (which is bad) or call pkg-config (which is good if your deps come from the system, but in this case they don't). But to make this use case work there would be a need to tell from "the outside" how to obtain which library and what their dependencies are. As far as I know there is no way to tell this to Cargo and any project that does its own thing with build.rs is automatically guaranteed to not work without code changes.

And that was the easy part

Suppose that in one of the libraries generates headers during compilation. An example would be Protocol Buffers. You can't start compiling any code that depends on those headers until they exist.  You could try to work around this by building one project at a time. This is the model of recursive Make and Visual Studio solution files. Its main downside is that it makes builds a lot slower. The interface specification would need to have the header information to get reliable builds.

Another thing about Protocol Buffers is that you need to build the compiler executable in order to compile proto definitions to sources. Like with headers there needs to be a way to tell the project to use the recently built protoc rather than trying to find it on its own. This also adds a new dependency between the projects. The interface specification needs to have all this information to get reliable builds.

All of this also needs to be in the extraction interface, too, in case library b generated headers or has an executable that should be used later on in the build.

The interface seems way too complicated...

Going through all the requirements you eventually find that in order to make things really, actually, work you need to serialise pretty much everything between the two projects. It would also need to be fully programming language agnostic. If we started the design work today it would take months or even years to get something working. Fortunately we don't necessarily have to.

This is exactly what Meson is.

All the work we have put in Meson for the past six years has been done to create a programming language independent way of describing builds and making them work together. As a side effect we have also created a reference implementation for such a build system that can be used to build code. But that is mostly incidental, the actual important thing is, and always has been, the description language. Every problem mentioned in this post has a solution in the Meson language and they have been used in production for year. Of course there would be problems and bugs found as the usage would grow but it's the same with every program. Bugs can be fixed.

What would it look like?

The build definition for library b that would support all the things discussed above would look like this (assuming the library has a pre-existing .h file and does not have a build.rs file):

project('libb', 'rust')

a_dep = dependency('a', fallback: ['a', 'a_dep'])

b = static_library('b', 'b.rs', dependencies: a_dep)

b_dep = declare_dependency(link_with: b, include_directories: '.')

And that's it. This is all you need to get full project interop (there may be some implementation bugs).

Best of all this can be implemented in 100% Rust without adversely affecting existing projects. Either Cargo can parse the toml definitions and export this file, or Cargo could be made to read Meson build definitions natively. As an extra bonus any Rust project that uses external dependencies could use this to get native and transparent pkgconfig support without needing to write a build.rs (obviously it would not work with Cargo proper unless support is added to Cargo to read Meson files).

Extra language advantage

There are many languages that want to become the next big systems programming language. Almost all of them advertise good C compatibility as a big selling point. There is not a single one, though, that has "good interoperability with C tooling" in their priority list. In fact it is often quite the opposite. Getting language specific build systems working within existing projects can be a massive pain because they have not been designed for that.

Thus if a language would take tooling compatibility as a priority and would produce Meson files, either as their build system's native format or through some sort of an export mechanism, it would have an ease of integration advantage over other languages. Libraries written in language X could be dropped inside other projects that build with Meson and they would basically work out of the box (again, modulo bugs).

So is this the One True Way of how things will go?

No. Of course not. There are many different ways to solve this problem and this is only one possible way (in my opinion a fairly good one, but obviously other people will have different opinions and priorities). The real solution will, at least in the short term, probably be something completely different. But in any case it should fulfil the requirements discussed above because you can't really combine two languages and get a usable and reliable combination without them.