Making Visual Studio compilers directly runnable from any shell (yes, even plain cmd.exe)

The Visual Studio compiler toolchain behaves in peculiar ways One of the weirdest is that you can't run the compiler from any shell. Instead you have to run the compiler either from a special, blessed shell that comes with VS (the most common are "x86 native tools shell" and "x64 native tools shell", there are also ARM shells as well as cross compilation shells) or by running a special bat file inside a pristine shell that sets things up. A commonly held misbelief is that using the VS compiler only requires setting PATH correctly. That is not true, it requires a bunch of other stuff as well (I'm not sure if all of that is even documented).

To anyone who has used unixy toolchains, this is maddening. The classic Unix approach is to have compiler binaries with unique names like a hypothetical armhf-linux-gcc-11 from any shell. Sadly this VS setup has been the status quo for decades now and it is unlikely to change. In fact, some times ago I had a discussion with a person from Microsoft where I told them about this problem and the response I got back was, effectively: "I don't understand what the problem is" followed by "just run the compiles from the correct shell".

So why is this a bad state of things then? There are two major issues. The first one is that you have to remember how every one of your build trees has been set up. If you accidentally run a compilation command using the wrong shell, the outcome is very undefined. This is the sort of things that happens all the time because human beings are terrible at remembering the states of complicated systems and specific actions that need to be taken depending on their state (as opposed to computers, which are exceptionally good at those things). The second niggle is that you can't have two different compilers active in the same shell at the same time. So if, for example, you are cross compiling and you need to build and run a tool as part of that compilation (e.g. Protobuf) then you can't do that with the command line VS tools. Dunno if it can be with solution files either.

Rolling up them sleeves

The best possible solution would be for Microsoft to provide compiler binaries that are standalone and parallel runnable with unique names like cl-14-x64.exe. This seems unlikely to happen in the near future so the only remaining option is to create them ourselves. At first this might seem infeasible but the problem breaks neatly down into two pieces:

• Introspect all changes that the vsenv setup bat file performs on the system.
• Generate a simple executable that does the same setup and then invokes cl.exe with the same command line arguments as were given to it.

The code that implements this can be found in this repository. Most of it was swiped from Meson VS autoactivator. When you run the script in a VS dev tools shell (it needs access to VS to compile the exe) you get a cl-x64.exe that you can then use from any shell. Here we use it to compile itself for the second time:

Downsides

Process invocation on Windows is not particularly fast and with this approach every compiler invocation becomes two process invocations. I don't know enough about Windows to know whether one could avoid that with dlopen trickery or the like.

For actual use you'd probably need to generate these wrappers for VS linkers too.

You have to regenerate the wrapper binary every time VS updates (at least for major releases, not sure about minor ones).

The end results has not been tested apart from simple tests. It is a PoC after all.

Using cppfront with Meson

Recently Herb Sutter published cppfront, which is an attempt to create C++ a new syntax to fix many issues that can't be changed in existing C++ because of backwards compatibility. Like with the original cfront compiler, cppfront works by parsing the "new syntax" C++ and transpiling it to "classic" C++, which is then compiled in the usual way. These kinds of source generators are fairly common (it is basically how Protobuf et al work) so let's look at how to add support for this in Meson. We are also going to download and build the cppfront compiler transparently.

Building the compiler

The first thing we need to do is to add Meson build definitions for cppfront. It's basically this one file:

project('cppfront', 'cpp', default_options: ['cpp_std=c++20'])

cppfront = executable('cppfront', 'source/cppfront.cpp',
  override_options: ['optimization=2'])

meson.override_find_program('cppfront', cppfront)
cpp2_dep = declare_dependency(include_directories: 'include')

The compiler itself is in a single source file so building it is simple. The only thing to note is that we override settings so it is always built with optimizations enabled. This is acceptable for this particular case because the end result is not used for development, only consumption. The more important bits for integration purposes are the last two lines where we define that from now on whenever someone does a find_program('cppfront') Meson does not do a system lookup for the binary but instead returns the just-built executable object instead. Code generated by cppfront requires a small amount of helper functionality, which is provided as a header-only library. The last line defines a dependency object that carries this information (basically just the include directory).

Building the program

The actual program is just a helloworld. The Meson definition needed to build it is this:

project('cpp2hello', 'cpp',

    default_options: ['cpp_std=c++20'])

cpp2_dep = dependency('cpp2')

cppfront = find_program('cppfront')

g = generator(cppfront,

  output: '@BASENAME@.cpp',

  arguments: ['@INPUT@', '-o', '@OUTPUT@']

  )

sources = g.process('sampleprog.cpp2')

executable('sampleprog', sources,

   dependencies: [cpp2_dep])

That's a bit more code but still fairly straightforward. First we get the cppfront program and the corresponding dependency object. Then we create a generator that translates cpp2 files to cpp files, give it some input and compile the result.

Gluing it all together

Each one of these is its own isolated repo (available here and here respectively). The simple thing would have been to put both of these in the same repository but that is very inconvenient. Instead we want to write the compiler setup once and use it from any other project. Thus we need some way of telling our app repository where to get the compiler. This is achieved with a wrap file:

[wrap-git]
directory=cppfront
url=https://github.com/jpakkane/cppfront
revision=main

[provide]
cpp2 = cpp2_dep
program_names = cppfront

Placing this in the consuming project's subprojects directory is all it takes. When you start the build and try to look up either the dependency or the executable name, Meson will see that they are provided by the referenced repo and will clone, configure and build it automatically:

The Meson build system

Version: 0.63.99

Source dir: /home/jpakkane/src/cpp2meson

Build dir: /home/jpakkane/src/cpp2meson/build

Build type: native build

Project name: cpp2hello

Project version: undefined

C++ compiler for the host machine: ccache c++ (gcc 11.2.0 "c++ (Ubuntu 11.2.0-19ubuntu1) 11.2.0")

C++ linker for the host machine: c++ ld.bfd 2.38

Host machine cpu family: x86_64

Host machine cpu: x86_64

Found pkg-config: /usr/bin/pkg-config (0.29.2)

Found CMake: /usr/bin/cmake (3.22.1)

Run-time dependency cpp2 found: NO (tried pkgconfig and cmake)

Looking for a fallback subproject for the dependency cpp2

Executing subproject cppfront 

cppfront| Project name: cppfront

cppfront| Project version: undefined

cppfront| C++ compiler for the host machine: ccache c++ (gcc 11.2.0 "c++ (Ubuntu 11.2.0-19ubuntu1) 11.2.0")

cppfront| C++ linker for the host machine: c++ ld.bfd 2.38

cppfront| Build targets in project: 1

cppfront| Subproject cppfront finished.

Dependency cpp2 from subproject subprojects/cppfront found: YES undefined

Program cppfront found: YES (overridden)

Build targets in project: 2

As you can tell from the logs, Meson first tries to find the dependencies from the system and only after it fails does it try to download them from the net. (This behaviour can be altered.) Now the code can be built and the end result run:

$ build/sampleprog

Cpp2 compilation is working.

The code has only been tested with GCC but in theory it should work with Clang and VS too.