A call for more downstream testing of Meson

As Meson gets more and more popular, the number of regressions also grows. This is an unvoidable fact of life. To minimize this effort we publish release candidates before the actual releases. Unfortunately not many people use these so many issues are not found until after the release (as happened with 0.60.0).

For this reason we'd like to ask more people to test these rcs on their systems. It's fairly straightforward.

Testing individual projects

If you have a CI that installs Meson using pip, this is easy. You can tell Pip to use prerelease versions with the --pre flag.

pip install --pre meson

If you use prebuilt images rather than reinstalling on every build, do update your images once a week. Meson releases happen traditionally on Sunday evenings European time.

Testing if you are a distro or similar

The release candidates are packaged and uploaded to Debian experimental, so if you can use those, it is the simplest solution. They are not uploaded to unstable as I was instructed not to do so because of breakage potential. If you are a Debian person and know that the above explanation is incorrect and that I should be doing something else, let me know so I can change how that is done.

If you have some different setup that has a full CI run (hopefully something smaller than a full Debian archive rebuild) then doing that with the rc version would be the best test.

If you don't have such a test suite, you'll probably want to set one up for other reasons as well. :)

Glyphtracer 2.0

Ages ago I wrote a simple GUI app called Glyphtracer to simplify the task of creating fonts from scanned images. It seems people are still using it. The app is written in Python 2 and Qt 4, so getting it running becomes harder and harder as time goes by.

Thus I spent a few hours porting it to Python3 and PyQt 5 and bumped the major version to 2.0. The program can be obtained from this Github repo.

Swappable faceplates for laptops

On the whole, laptops are boring. There are three different types.

1. Apple aluminium laptops that all look the same.
2. Other manufacturer's laptops that try to look like Apple laptops that look all the same.
3. RGB led studded gamer laptops that all look the same.

There is a cottage industry that manufactures stickers that are the size of the entire laptop but those are inconvenient and not particularly slick. You only have one chance to apply and any misalignments or air bubbles are going to be there forever.

But what if some laptop manufacturer chose to do things differently and designed their laptops so that the entire face plate were detachable and replaceable? This would allow everybody to customize their own laptops in exactly the way they want to (this should naturally extend to the inner keyboard plate, but we'll ignore that for now). For example you could create a laptop with the exact Pantone color that you want rather than what the laptop manufacturer saw fit to give you.

Or maybe you'd like to have a laptop cover which lights up not the manufacturer's branding but instead the logo of your favourite band?

Or perhaps you are a fan of classic muscle cars?

The thing that separates these covers from what we currently have is that they would not be pictures of things or stickers. The racing stripe cover could be manufactured in exactly the same way as real car hoods. It would shine like real metal. It would feel different when touched. You could, one imagines, polish it with real car wax. It would be the real thing. Being able to create experimental things in isolation makes all sorts of experimentation possible so you could try holograms, laser engravings, embossings, unusual metals and all sorts of funky stuff easily. In case of failure the outcome is just a lost cover plate you can toss into the recycling bin rather than a very expensive laptop that is either unusably ugly or flat out broken.

But wait, there's more. Corporate PR departments should be delighted with this. Currently whenever people do presentations in conferences their laptops are clearly visible and advertise the manufacturer. The same goes for sales people visiting customers (well, eventually, once Covid 19 passes) and so on. Suppose you could have this instead:

Suddenly you have reclaimed a ton of prime advertising real estate that can be used for brand promotion (or something like that, not really a PR person so I'm not intimately familiar with the terminology). If you are a sufficiently large customer and order thousands of laptops at a time, it might be worth it to get all the custom plates fitted at the main factory during assembly. Providing this service would also increase the manufacturer's profit margin.

Hercule Poirot and the Mystery Box, as written by JJ Abrams

Outside, the scorching heat of the British summer could not have made a starker contrast with the icy cold atmosphere of the main living room of Lord Ellington's mansion. All fifteen people sitting around it were staring at the plump Belgian private inspector, who seemed to take great pleasure in waxing his mustache and letting the audience simmer in their own sweat. Finally he spoke.

"You may have wondered why I have called all you here. It has to do with the untimely death of Lady Sybill. We have reason to believe that the three dozen stab wounds in her back were not an accident but that she was in fact ..."

He paused for effect.

"... murdered."

Madame Smith shrieked in terror and dropped her tea cup, which shattered into a million trillion pieces, each of which glittered with the sparkle of a thousand suns. Which was strange, since the curtains were closed and there were no sources of direct light in the room.

"Exactly, mon frére" he said to her, even though his mother tongue was French and he should have known not to call a woman "my brother". And also to spell his sentences properly. But who cares about trivial details such as these when there is a mystery afoot?

"And furthermore, I'm happy to tell you all that our investigation on this has finally come to a conclusion."

Poirot took off his monocle, polished it and then put it back on.

"Oui, we can finally reveal that the identity of the murderer ... will remain unknown forever. Even with our plethora of clues, we could not determine what actually happened. Whoever the murderer was, they got away scott free as we are shutting down the investigation. Mon dieu, that makes this our seventy-sixth unsuccessful murder investigation in a row!"

The audience in the salon seemed perplexed, so captain Hastings chose to interject.

"It could also be space aliens!"

"My money is on an interdimensional time travel accident" countered Inspector Japp with the calm voice of an experienced police officer.

For a while, the room was enveloped in silence. Then the house was blown up by nazis.

Looking at building O3DE with Meson, part II

After the first post, some more time was spent on building O3DE with Meson. This is the second and most likely last post on the subject. Currently the repository builds all of AzCore basic code and a notable chunk of its Qt code. Tests are not built and there are some caveats on the existing code, which will be discussed below. The rest of the conversion would most likely be just more of the same and would probably not provide all that much new things to tackle.

Code parts and dependencies

Like most projects, the code is split into several independent modules like core, testing, various frameworks and so on. The way Meson is designed is that you traverse the source tree one directory at a time. You enter it, do something, possibly recurse into subdirectories and then exit it. Once exited you can never again return to the directory. This imposes some extra limitations on project structure, such as making circular dependencies impossible, but also makes it more readable.

This is almost always what you want. However there is one exception that many projects have: the lowest layer library has no internal dependencies, the unit testing library uses that library and the tests for the core library use the unit testing library. This is not a circular dependency as such, but if the unit tests are defined in the same subdir as the core library, this causes problems as you can't return to it. This needs to be broken in some way, like the following:

subdir('AzCore')
subdir('AzTest')
subdir('AzCore/tests')

Code generation

Most large projects have a code generator. O3DE is no exception. Its code generator is called AutoGen and it's a Python script that expands XML using Jinja templates. What is strange is that it is only used in three places, only one of which is in the core code. Further, if you look at the actual XML source file it only has a few definitions. This seems like a heavy weighted way to go about it. Maybe someone could summon Jason Turner to constexrpify it to get rid of this codegen.

This part is not converted, I just commented out the bits that were using it.

Extra dependencies

There are several other dependencies used that seem superfluous. As an example the code uses a standalone library for MD5, but it also uses OpenSSL, which provides an MD5 implementation. As for XML parsers, there are three, RapidXML, Expat and the one from Qt (though the latter is only used in the editor).

Editor GUI

Almost all major game engines seem to write their own GUI toolkits from scratch. Therefore it was a bit surprising to find out that O3DE has gone all-in on Qt. This makes it easy to use Meson's builtin Qt 5 support, though it is not without some teething issues. First of all the code has been set up so that each .cpp file #includes the moc file generated from its header:

#include "Components/moc_DockBarButton.cpp"

Meson does things differently and builds the moc files automatically so users don't have to do things like this. They are also written in a different directory than what the existing configuration uses so this include could not work, the path is incorrect. This #include could be removed altogether, but since you probably need to support both at the same time (due to, for example, a transition period) then you'd need to do something like this:

#ifndef MESON_BUILD

#include "Components/moc_DockBarButton.cpp"

#endif

What is more unfortunate is that the code uses Qt internal headers. For some reason or another I could not make them work properly as there were missing private symbols when linking. I suspect that this is because distro Qt libraries have hidden those symbols so they are not exported. As above I just commented these out.

The bigger problem is that O3DE seems to have a custom patches in their version. At least it refers to style enum values that do not exist. Googling for the exact string produces zero relevant matches. If this is the case then the editor can not be used with official Qt releases. Further, if said patches exist, then they would need to be provided to the public as per the LGPL, since the project is providing prebuilt dependency binaries. As mentioned in the first blog post, the project does not provide original sources for their patched dependencies or, if they do, finding them is not particularly easy.

What next?

Probably nothing. It is unlikely that upstream would switch from CMake to Meson so converting more of the code would not be particularly beneficial. The point of this experiment was to see if Meson could compile O3DE. The answer for that is yes, there have not been any major obstacles. The second was to see if the external dependencies could be provided via Meson's Wrap mechanism. This is also true, with the possible exception of Qt.

The next interesting step would be to build the code on multiple platforms. The biggest hurdle here is the dependency on OpenSSL. Compiling it yourself is a bear, and there is not a Wrap for it yet. However once this merge request is merged, then you should be able to build OpenSSL as a Meson subproject transparently. Then you could build the core fully from source on any platform.

How much effort would it take to convert OpenSSL's Perl source code generators to Python?

There is an ongoing discussion to write Meson build definitions to OpenSSL so it can be added to the WrapDB and built transparently as a subproject. One major issue is that OpenSSL generates a lot of assembly during build time with Perl. Having a Perl dependency would be bad, but shipping pregenerated source files would also be bad. Having "some pregenerated asm" that comes from "somewhere" would understandably be bad in a crypto library.

The obvious third option would be to convert the generator script from Perl to Python. This is not the first time this has been proposed and the counterargument has always been that said conversion would take an unreasonable amount of effort and could never be done. Since nobody has tried to do the conversion we don't really know whether that claim is accurate or not. I converted the x86_64 AES asm generator to see how much work it would actually take. The code is here.

The code setup

The asm generator has two different levels of generators. First a Perl script generates asm in a specific format and after that a second Perl script converts it to a different asm type if needed (Nasm, At&T, Intel etc). The first script can be found here and the second one is here. In this test only the first script was converted.

At first sight the task seems formidable. The generator script is 2927 lines of code. Interestingly the asm file it outputs is only 2679 lines. Thus the script is not only larger than its output, it is also less understandable than the thing it generates both because it is mishmash of text processing operations and because it is written in Perl.

Once you get over the original hump, things do get easier. Basically you have a lot of strings with statements that look like this:

movzb `&lo("$s2")`,$acc0

This means that the string needs to be evaluated so that $s2 and $acc0 are replaced with the values of variables s2 and acc0. The backticks mean that you have to then call the lo function with the given value and substitute that in the output string. This is very perlistic and until recently would not have been straightforward to do in Python. Fortunately now we have f-strings so that becomes simply:

f'movzb {lo(s2)},{acc0}'

With that worked out the actual problem is no longer hard, only laborious. First you replace all the Perl multiline strings with Python equivalents, then change the function declarations from something like this:

sub enctransform_ref()
{ my $sn = shift;

to this:

def enctransform_ref(sn):

and then it's a matter of repeated search-and-replaces for all the replacement variables.

The absolute best thing about this is that it is almost trivial to verify that the conversion has been done without errors. First you run the original script and store the generated assembly. Then you run your own script and compare the output. If they are not identical then you know exactly where the bug is. It's like having a unit tests for every print statement for the program.

How long did it take?

From scratch the conversion took less than a day. Once you know how it's done a similar conversion would take maybe a few hours. The asm type converter script seems more complicated so would probably take longer.

A reasonable port would contain these conversions for the most popular algorithms to the most popular CPU architectures (x86, x86_64, arm, aarch64). It would require a notable amount of work but it should be measured in days rather than months or years. I did browse through some of the other asm files and it seems that they have generators that work in quite different ways. Converting them might take more or less work, but probably it would still be within an order of magnitude.

A quick look at the O3DE game engine and building it with Meson

Earlier today I livestreamed what it would take to build a small part of the recently open sourced O3DE game engine. The attempt did not get very far, so here is a followup. It should not be considered exhaustive in any way, it is literally just me poking the code for a few hours and writing down what was discovered.

Size

The build system consists of 56 thousand lines of CMake.

Name prefixes

The code has a lot of different naming convention and preserved prefixes. These include "Cry" which seem to be the oldest and come from the original CryEngine. The most common one is "az", which probably stands for Amazon. The "ly" prefix probably stands for "Lumberyard" which was the engine's prior name. Finally there is "PAL", which is not really explained but seems to stand for "platform abstraction layer" or something similar.

Compiling

The code can be obtained from the upstream repo. There's not much more you can do with it since it does not actually build on Linux (tested on latest Fedora) but instead errors out with a non-helpful CMake error message.

When I finally managed to compile something, the console practically drowned in compiler warnings. These ranged from the common (missing virtual destructors in interface classes) to the bizarre (superfluous pragma pops that come from "somewhere" due to macros).

Compiler support

The code explicitly only supports Visual Studio and Clang. It errors out when trying to build with GCC. Looking through the code it seems like it is mostly a case of adding some defines. I tried that but pretty quickly ran into page-long error messages. A person with more knowledge of the inner workings of GCC could probably make it work with moderate effort.

Stdlib reimplementations

O3DE is based on CryEngine, which predates C++ 11. One place where this shows up is that rather than using threading functionality in the standard library they have their own thread, mutex, etc classes that are implemented with either pthread or Windows threads. There may be some specific use cases (thread affinity?) why you'd need to scoop to plain primitives but otherwise this seem like legacy stuff nobody has gotten around to cleaning.

Yes, there is a string class. Several, in fact. But you already knew that.

Dependencies

This is where things get weird. The code uses, for example, RapidXML and RapidJSON. For some reason I could not get them to work even though I used the exact same versions that are listed in the CMake definition. After a fair bit of head scratching things eventually became clear. For example the system has its own header for RapidXML called rapidxml.h whose contents are roughly the following:

#define RAPIDXML_SKIP_AZCORE_ERROR

// the intention is that you only include the customized version

// of rapidXML through this header, so that

// you can override behavior here.

#include <rapidxml/rapidxml.h>

Upstream does not provide its header in a rapidxml subdirectory, it is under include. The same happens when the header is installed on the system. Thus the include as given can not work. Even more importantly, the upstream header is not named rapidxml.h but instead rapidxml.hpp.

It turns out that O3DE has its own dependency system which takes upstream source, makes arbitrary changes to it, builds it and then is provided as a "package" which is basically a zip file with headers and prebuilt static libs. These are downloaded from not-at-all-suspicous-looking URLs like this one. What changes are done to these packages is not readily apparent. There are two different repos with info but they don't seem to have everything.

When using external libraries like this there are two typically two choices. Either you patch the original to "fit in" with the rest of your code or you can write a very small adapter wrapper. This project does both and with preprocessor macros no less.

The whole dependency system seems to basically be a reimplementation of Conan using pure CMake. If that sentence on its own does not make cold sweat run down your back then let it be noted that one of the dependencies obtained in this way is OpenSSL.

The way the system is set up prevents you from building the project using system dependencies. This includes Qt as the editor GUI is written with it. Neither can you build the entire project from source yourself because the existing source only works with its own special prebuilt libraries and the changes applied do not seem to be readily available as patches.

Most of this is probably because CryEngine was originally written for internal use on Windows only. This sort of an approach works for that use case but not all that well for a multiplatform open source project.

Get the code

My experimental port that compiles only one static library (AzCore) can be downloaded from this Github repo. It still only supports Clang but adding GCC support should be straightforward as you don't need to battle the build mechanism at least.