- The build system creates a Ninja file as usual
- It sets up a dependency so that every compilation job depends on a prescan step.
- The scanner goes through all source files (using compilation_commands.json), determines module interdependencies and writes this out to a file in a format that Ninja understands.
- After the scan step, Ninja will load the file and use it to execute commands in the correct order.
Can we do it regardless? Yes we can!
Enter self-modifying build system code
The basic approach is simple
- Write a Ninja file as usual, but make all the top level commands (or, for this test, only all) run a secret internal command.
- The command will do the scanning, and change the Ninja file on the fly, rewriting it to have the module dependency information.
- Invoke Ninja on the new file giving it a secret target name that runs the actual build.
- Build proceeds as usual.
- incremental builds probably won't work
- the resulting binary never finishes (it is running a job with exponential complexity)
- it does not work on any other project than the demo one (but it should be fixable)
- the dependencies are on object files rather than module BMI files due to a Ninja limitation
- module dep info is not cached, all files are fully rescanned every time
- the scanner is not reliable, it does the equivalent of dumb regex parsing
- any and all things may break at any time and if they do you get to keep both pieces
Is this the way C++ module building will work?
Probably not, because there is one major use case that this approach (or indeed any content scanning approach) does not support: code generation. Scanning assumes that all source code is available at the same time but if you generate source code on the fly, this is not the case. There would need to be some mechanism of making Ninja invoke the scanner anew every time source files appear and such a mechanism does not exist as far as I know. Even if it does there is a lot of state to transfer between Ninja and the scanner to ensure both reliable and minimal dependency scans.
There are alternative approaches one can take to avoid the need for scanning completely, but they have their own downsides.