Proposal for target-private directories for compilers

One of the greatest strengths of the classical C compiler model is that all compile jobs are fully isolated. This means that they can be run perfectly in parallel. This same feature is also one of its greatest weaknesses. There are no ways for individual compile jobs to communicate with each other even if they wanted to. This could be useful for things like caching. As an example a compiler might transparently create "precompiled headers" of sorts during one compilation and use them in other compilations if needed. This might also be useful for languages that require scanning steps before building such as Fortran and C++ using modules.

This is not idle speculation. Clang's thinLTO does use caching to speed up incremental builds. Since there is no existing standard for this, they did the usual thing and created a new compiler flag for specifying the location of the cache directory. Or, to be precise, they created four of them:

• gold (as of LLVM 4.0): -Wl,-plugin-opt,cache-dir=/path/to/cache
• ld64 (support in clang 3.9 and Xcode 8): -Wl,-cache_path_lto,/path/to/cache
• ELF lld (as of LLVM 5.0): -Wl,--thinlto-cache-dir=/path/to/cache
• COFF lld-link (as of LLVM 6.0): /lldltocache:/path/to/cache

For one option this is tedious but for many it becomes completely unworkable. Clearly something better is needed.

The basic idea: each build target gets its own private directory

Basically what one should be able to do is this:

gcc -c -o foo/bar.o bar.c -fprivate-dir=some_path

The private directory would have the following properties:

• The build system guarantees that it is set to the same value for all compilations of a single target (executable, shared library, static library, etc)
• Every build target gets its own unique private directory
• The contents of the directory may persist over successive invocations (i.e. its contents may be deleted at any time, but most of the time won't be)
• The compiler can create anything it wants in the private dir but should also tolerate other usages (typically you'd also want to put the target's object files in this dir)
• The contents of the private dir are transitory, they have no backwards or forwards compatibility guarantees. Any compiler update would invalidate all files.

If, for example, compilers wanted to create pipes or Unix domain sockets in the private dir for communicating between compiler instances, they could do that behind the scenes.

Work needed for tooling

Meson and CMake already to pretty much exactly this as they store object files in special directories. I don't know enough about Autotools to know how much work it would be, though it does have the concept of higher level build targts. Handwritten Makefiles would need to be tweaked by hand as with every change. Visual Studio solutions are already split up to per-target project files so adding new flags there should be fairly simple.

The best part is that this change would be fully backwards compatible. If the private dir argument is not used, the compilers would behave in exactly the same way they do now.