Making Visual Studio compilers directly runnable from any shell (yes, even plain cmd.exe)

The Visual Studio compiler toolchain behaves in peculiar ways One of the weirdest is that you can't run the compiler from any shell. Instead you have to run the compiler either from a special, blessed shell that comes with VS (the most common are "x86 native tools shell" and "x64 native tools shell", there are also ARM shells as well as cross compilation shells) or by running a special bat file inside a pristine shell that sets things up. A commonly held misbelief is that using the VS compiler only requires setting PATH correctly. That is not true, it requires a bunch of other stuff as well (I'm not sure if all of that is even documented).

To anyone who has used unixy toolchains, this is maddening. The classic Unix approach is to have compiler binaries with unique names like a hypothetical armhf-linux-gcc-11 from any shell. Sadly this VS setup has been the status quo for decades now and it is unlikely to change. In fact, some times ago I had a discussion with a person from Microsoft where I told them about this problem and the response I got back was, effectively: "I don't understand what the problem is" followed by "just run the compiles from the correct shell".

So why is this a bad state of things then? There are two major issues. The first one is that you have to remember how every one of your build trees has been set up. If you accidentally run a compilation command using the wrong shell, the outcome is very undefined. This is the sort of things that happens all the time because human beings are terrible at remembering the states of complicated systems and specific actions that need to be taken depending on their state (as opposed to computers, which are exceptionally good at those things). The second niggle is that you can't have two different compilers active in the same shell at the same time. So if, for example, you are cross compiling and you need to build and run a tool as part of that compilation (e.g. Protobuf) then you can't do that with the command line VS tools. Dunno if it can be with solution files either.

Rolling up them sleeves

The best possible solution would be for Microsoft to provide compiler binaries that are standalone and parallel runnable with unique names like cl-14-x64.exe. This seems unlikely to happen in the near future so the only remaining option is to create them ourselves. At first this might seem infeasible but the problem breaks neatly down into two pieces:

• Introspect all changes that the vsenv setup bat file performs on the system.
• Generate a simple executable that does the same setup and then invokes cl.exe with the same command line arguments as were given to it.

The code that implements this can be found in this repository. Most of it was swiped from Meson VS autoactivator. When you run the script in a VS dev tools shell (it needs access to VS to compile the exe) you get a cl-x64.exe that you can then use from any shell. Here we use it to compile itself for the second time:

Downsides

Process invocation on Windows is not particularly fast and with this approach every compiler invocation becomes two process invocations. I don't know enough about Windows to know whether one could avoid that with dlopen trickery or the like.

For actual use you'd probably need to generate these wrappers for VS linkers too.

You have to regenerate the wrapper binary every time VS updates (at least for major releases, not sure about minor ones).

The end results has not been tested apart from simple tests. It is a PoC after all.