Tuesday, July 20, 2021

A quick look at the O3DE game engine and building it with Meson

Earlier today I livestreamed what it would take to build a small part of the recently open sourced O3DE game engine. The attempt did not get very far, so here is a followup. It should not be considered exhaustive in any way, it is literally just me poking the code for a few hours and writing down what was discovered.

Size

The build system consists of 56 thousand lines of CMake.

Name prefixes

The code has a lot of different naming convention and preserved prefixes. These include "Cry" which seem to be the oldest and come from the original CryEngine. The most common one is "az", which probably stands for Amazon. The "ly" prefix probably stands for "Lumberyard" which was the engine's prior name. Finally there is "PAL", which is not really explained but seems to stand for "platform abstraction layer" or something similar.

Compiling

The code can be obtained from the upstream repo. There's not much more you can do with it since it does not actually build on Linux (tested on latest Fedora) but instead errors out with a non-helpful CMake error message.

When I finally managed to compile something, the console practically drowned in compiler warnings. These ranged from the common (missing virtual destructors in interface classes) to the bizarre (superfluous pragma pops that come from "somewhere" due to macros).

Compiler support

The code explicitly only supports Visual Studio and Clang. It errors out when trying to build with GCC. Looking through the code it seems like it is mostly a case of adding some defines. I tried that but pretty quickly ran into page-long error messages. A person with more knowledge of the inner workings of GCC could probably make it work with moderate effort.

Stdlib reimplementations

O3DE is based on CryEngine, which predates C++ 11. One place where this shows up is that rather than using threading functionality in the standard library they have their own thread, mutex, etc classes that are implemented with either pthread or Windows threads. There may be some specific use cases (thread affinity?) why you'd need to scoop to plain primitives but otherwise this seem like legacy stuff nobody has gotten around to cleaning.

Yes, there is a string class. Several, in fact. But you already knew that.

Dependencies

This is where things get weird. The code uses, for example, RapidXML and RapidJSON. For some reason I could not get them to work even though I used the exact same versions that are listed in the CMake definition. After a fair bit of head scratching things eventually became clear. For example the system has its own header for RapidXML called rapidxml.h whose contents are roughly the following:

#define RAPIDXML_SKIP_AZCORE_ERROR

// the intention is that you only include the customized version
// of rapidXML through this header, so that
// you can override behavior here.
#include <rapidxml/rapidxml.h>

Upstream does not provide its header in a rapidxml subdirectory, it is under include. The same happens when the header is installed on the system. Thus the include as given can not work. Even more importantly, the upstream header is not named rapidxml.h but instead rapidxml.hpp.

It turns out that O3DE has its own dependency system which takes upstream source, makes arbitrary changes to it, builds it and then is provided as a "package" which is basically a zip file with headers and prebuilt static libs. These are downloaded from not-at-all-suspicous-looking URLs like this one. What changes are done to these packages is not readily apparent. There are two different repos with info but they don't seem to have everything.

When using external libraries like this there are two typically two choices. Either you patch the original to "fit in" with the rest of your code or you can write a very small adapter wrapper. This project does both and with preprocessor macros no less.

The whole dependency system seems to basically be a reimplementation of Conan using pure CMake. If that sentence on its own does not make cold sweat run down your back then let it be noted that one of the dependencies obtained in this way is OpenSSL.

The way the system is set up prevents you from building the project using system dependencies. This includes Qt as the editor GUI is written with it. Neither can you build the entire project from source yourself because the existing source only works with its own special prebuilt libraries and the changes applied do not seem to be readily available as patches.

Most of this is probably because CryEngine was originally written for internal use on Windows only. This sort of an approach works for that use case but not all that well for a multiplatform open source project.

Get the code

My experimental port that compiles only one static library (AzCore) can be downloaded from this Github repo. It still only supports Clang but adding GCC support should be straightforward as you don't need to battle the build mechanism at least.

Thursday, July 8, 2021

A followup to the Refterm blog post

My previous blog post was about some measurements I made to Refterm. It got talked about on certain places on the net from where it got to Twitter. Then Twitter did the thing it always does, which is to make everything terrible. For example I got dozens of comments saying that I was incompetent, an idiot, a troll and even a Microsoft employee. All comments on this blog are manually screened but this was the only time when I just had to block them all. Going through those replies  seemed to indicate that absolutely everyone involved [1] had bad communication and also misunderstood most other people involved. Thus I felt compelled to write a followup explaining what the blog post was and was not about. Hopefully this will bring the discussion down to a more civilized footing.

What we agree on

Let's start by writing down all the things we agree on.

  1. The current Windows terminal is slow.
  2. It can be made faster.
  3. A GPU-based renderer (such as the one in Refterm) can render terminal text hundreds of times faster than the current implementation in Windows terminal.
Note that even the people working on Microsoft Terminal acknowledged all of these to be true before any code on Refterm had been written. From what I can tell, #3 is what Refterm set out to prove and it was successfull at it.

So what's the issue then?

Once the code was out many people started making these kinds of statements.

Windows terminal should switch to using this method because it is obviously superior and not doing that is just laziness and excuses.

Now, and this is important, the original people who worked on Refterm never made these kinds of claims. They did not! And further, I never claimed that they did. Other people ("the talking heads on the Internet") made those claims and then mental misattribution took over. This is unfortunate but sadly almost inevitable whenever these kinds of debates happen. That then leads to the obvious follow up question:

Could the rendering mechanism used in Refterm be put in Windows terminal proper? If not, why not? If yes, what would the outcome be like and would the code need changing?

This is what my original blog post was about. Since this was outside of the original project's design goals I should have stated the goals out explicitly. I did not, and that is a flaw on my part, sorry about that.

The problems with prototypes

Implementing a simple prototype of an existing program (or a part of it), achieving great success and then extrapolating from that to the whole program (and to reiterate: the original Refterm authors did not do this speculation, other people did) has a well known pitfall. I could write an entire blog post about it. Fortunately I don't have to since Matthew Garrett already wrote one. I recommend that everyone reads that before continuing with this post.

The tl/dr version is that when you bring a protype up to sufficient feature parity with an existing implementation you will encounter unexpected problems. The algorithms or approaches you took might not be feasible. You might need to add a bunch of functionality that you never considered or had even heard of. Until you have the entire implementation you don't know whether you approach will work. In fact you can't know it. Anyone who claims to know is lying, either to others or to themselves. (Reminder again: the Refterm authors did not make these kinds of estimates.)

We can try to come up with some of the obstacles and problems one could have when moving the prototype implementation into a real one and then examine those. They can't prove fitness but they can reveal un-fitness. The points discussed in the blog post were just some that I came up with. There are undoubtedly many others.

Resource usage

Let's start this by acknowledging a notable flaw in the original post [2]. When evaluating memory usage the post only compared it against other types of apps, not other terminals. I ran some new measurements by starting both the Windows cmd.exe shell as well as the Git-Scm's MSYS2 terminal, running a few simple commands and looking at memory consumption with the Task Manager. Refterm took 350 MB of ram, MSYS2 took 4 MB and cmd.exe took 7 MB.

People really love their terminals. I have seen setups where a single developer has 10+ different terminals open at the same time all as different processes (with several tabs each). So even if 300 MB of ram usage for a single app would be fine, using 3 GB of ram in this case would not be. Thus one would either need to dramatically reduce memory usage or have something like a shared glyph cache between all the various processes. That requires shared GPU resources, some sort of an IPC communication mechanism, multiprocess cache invalidation and all other fun stuff (or that is my understanding at least as a Windows and GPU neophyte, if there is a simpler way do let me know).

This piece of information is useful and important on its own. It gives new information and understanding of what the code does and does not do.

A retort to this that was actually made by Refterm developers was that "there are variables and knobs in the code you can tweak to affect the performance". To this I say: no. The first tests should always be done with the exact setup the code ships with. There are two reasons for this. First of all, that makes experiments made by different people directly comparable with each other. Secondly, the original author(s) know the code best and thus it makes sense to choose those parameter values that they went with. 

Code layout and nonstandardness

Let's start again with the thing we all agree on:

For your own projects you can choose whatever code layout, build system, organization and so on that you want. Do whatever works best for you and don't let anyone tell you otherwise!

Things get more complicated when you start including other people, especially outside your own circle of devs. An open source project is a typical example. An anonymous commenter told me the following:

This is also the simplest possible code structure, very simple to work with for new contributors.

This sentence is interesting in that it is both true and not true. If you have a person who has no prior programming knowledge then yes, the layout is simplest possible and easy to get started with. On the other hand if the potential contributor is already accustomed to the "standard way of setting up a project" then things change and the nonstandard layout can get confusing [3] and can be a barrier to entry for new people. This is the nature of teamwork: sometimes it might make sense to do the thing that is inconvenient for you personally for the benefit of others. Sometimes it does not make sense. Like most things in life being nonstandard is not an absolute thing, it has its advantages but also disadvantages.

I actually encountered a technical disadvantage caused by this. I wanted to compile and run Refterm under Address Sanitizer, which is a really great tool for finding bugs. Asan is integrated into the latest VS and all you need to do is to add /fsanitize=address flags to the compiler to use it. This does not work for Refterm but instead leads to a bunch of linker errors. The Asan library depends on the C runtime and Refterm is explicitly set up not to use it. It took me a fair bit of time to work out that the way to get it working is to go through the code and replace the WinMainCRTStartup function with a "normal" WinMain function and then the linker would do the right thing [4].

That SIMD memcpy thing

I pondered for a while whether I should mention the memcpy thing and now I really wish I hadn't. But not for the reasons you might think.

The big blunder I did was to mention SIMD by name, because the issue was not really about SIMD. The compiler does convert the loop to SIMD automatically. I don't have a good reference, but I have been told that Google devs have measured that 1% of all CPU usage over their entire fleet of computers is spent on memcpy. They have spent massive amounts of time and resources on improving its performance. At least as late as 2013, performance optimizing memcpy was still subject to fundamental research (or software patents at least). For reference here is the the code for the glibc version of memcpy, which seems to be doing some special tricks.

If this is the case and the VS stdlib provides a really fast memcpy then rolling your own does cause a performance hit (though in this particular case the difference is probably minimal, possibly even lost in the noise). On the other hand it might be that VS can already optimize the simple version to the optimal code in which case the outcome is the same for both approaches. I don't know what actually happens and finding out for sure would require its own set of tests and benchmarks.

Concluding and a word about blog comments

That was a very long post and it did not even go through all the things I had in mind. If you have any comments, feel free to post them below, but note the following:

  • All comments are prescreened and only appear after manually approved.
  • Any comments that contain insults, whining, offensive tone or any other such thing will be trashed regardless of its other merits
  • The same goes for any other post whose contents makes it obvious that the commenter has not read the whole text but is just lashing out.

[1] Yes, this includes me. It most likely includes you as well.

[2] Thanks to an anonymous commenter for pointing this out.

[3] I can't speak for others, bu it was for me.

[4] There may have been other steps, but this was the crucial one.

Monday, July 5, 2021

Looking at the performance of Refterm

Recently a known strong-opinion-holder Casey Muratori wrote a reference implementation for a new fast terminal renderer.  The goal is (as far as I can tell) to implement the same functionality as existing terminals using a lot less resources. The code is here and according to the readme it supports, for example:

  • All of Unicode, including combining characters and right-to-left text like Arabic
  • Line wrapping
  • Reflowing line wrapping on terminal resize
The tests they perform show that the new terminal works several orders of magnitude faster than the default terminals in Windows. Seems nice, so let's do a simple code review to see how it actually stacks up.

Code setup

The code is arranged in a very non-standard way. There are two main source files, one C and one C++ that are built with a BAT file. Those files first #define a bunch of stuff and then #include all other source code files (not header files, source files) directly. This is very old school and against pretty much any recommended best practice for C. This works will for single-person projects but is a big hurdle for any new contributors.

The build uses the /GS- /Gs999999set command line arguments, which disable security features. This seems ill-advised for a terminal application, whose main job is to parse untrusted input. All libraries used are not defined in the build file but instead as pragmas inside the source files. The program also does not link the C library and because of this has its own simple implementations of memcpy and memset. This means you don't get the SIMD-optimized versions from the stdlib (the performance impact of this was not tested).

Resource usage

Resource consumption was measured by checking out the code, building it with the bat as instructed by upstream, starting the program and letting it idle. This is how it appears in Windows' task manager.

The app uses 0.5% of CPU and a whopping 14% of GPU just to display a blinking cursor. This could be said to be not particularly resource efficient. This is probably due to the fact that there is no rate limiter (or VSYNC) so the app just spams the system all the time. The true resource usage can't be meaningfully compared until this is fixed.

What can be measured, though, is memory usage. As can be seen in the image [1] the Refterm application uses 351 MB of memory when idle (the test war run using a 4k monitor). Based on discussions on the Internet, an acceptable amount of memory usage for a terminal is around 10-20 MB. Refterm uses 10x as much. In fact, as you can tell, running two instances of Refterm takes more memory than a fully blown Firefox with tens of open tabs. For comparison I also tested the Spotify app which is implemented in Electron. When playing music it only took ~150 MB, less than half of an idling Refterm.

Reliability

A terminal widget is a high risk app because it has to parse a lot of untrusted text. Thus it is very important that it tolerates malformed and malicious input. A simple test is to start Refterm and then run splat refterm_debug_msvc.pdb in Refterm. This basically dumps 1.3 MB of binary data to the terminal. This causes Refterm to immediately freeze and take 100% CPU. The app window can not be closed and can only be killed via the task manager.

Conclusions

In its current form Refterm can, at most, be seen as a proof of concept which can not be reasonably compared against a full-featured terminal renderer. It is neither memory-efficient nor reliable against malformed input.

[1] I don't know much about Windows development, so I don't know how representative this number is of the "real" resource usage. At least on Linux the output of top should be taken with a grain of salt. I also tried VS's profiler and it claimed that the app took over 450 MB of ram.

Update 2021/7/7

This blog post got linked to from places which caused a flood of mostly nonproductive and toxic comments. Because of this no further comments are allowed for this post. Sorry.

Friday, June 11, 2021

Typesetting a full book part II, Scribus

Some time ago I wrote a blog post on what it's like to typeset an entire book using nothing but LibreOffice. One of the comments mentioned that LO does not do a great job of aligning text. This is again probably because it needs to copy MS Word's behaviour, which means greedy line splitting. Supposedly Scribus does this a lot better, but the only way to be really sure was to typeset the whole text with Scribus. So that's what I did (using the latest 1.5 release from Flathub).

Workflow for Scribus

Every program has the things it is good for and things it's not that good for. Scribus' strengths lie in producing output with fairly short pieces of text with precise layout requirements, especially if there are many images. A traditional "single flow of text" is not that, so there are some things you need to plan for.

First of all, a Scribus document should not be created until the text is almost completely finished. Doing big changes (like adding text to existing chapters, changing physical page size etc) can become quite tedious. Scribus also does not do long pieces of text particularly smootly. I tried loading all 350is pages to a single linked frame sequence. It sort of worked, but things got quite laggy quite quickly. Eventually I converged on a layout where every chapter was its own set of linked frames. The text was imported directly from LO files that held one chapter each. The original had just one big LO file, so I had to split it up by hand for the import. If the original had been done with master documents, this would have been simpler.

The table of contents had to be done by hand again. Scribus has support for tables, but they could not be used, because tables drew outlines around each cell and I could not find a way to switch that off. Websearching found several pages with info, but none of them worked. It also turns out that you can not add page references to table cells, only to text frames. No, I don't know why either. The option was greyed out in the menus and trying to sneakily copypaste a page reference from a text frame to a table caused a segfault.

Issues discovered

While LO was surprisingly bug free, Scribus was less so and I encountered many bugs and baffling missing features, such as:
  • Scribus would sometimes create empty text frames far outside the document (i.e. to page 600 on a 300 page document)
  • Text frames got a strange empty character at their end which would cause text overflow warnings, deleting it did not help as the empty characters kept reappearing
  • Adding a page reference to an anchor point would always link to the page where the linked frame sequence started, not where the anchor was placed
  • Text is not hyphenated automatically, only by selecting a text frame and then selecting extras > hyphenate text in the main menu, one would imagine hyphenation being a paragraph style property instead
  • I managed to create an anchor point that does not exist anywhere except the mark list, but deleting it leads to an immediate segfault
None of these obstacles were insurmountable, but they made for a nonsmooth experience. Eventually the work was done and here is how they compare (LO on the left, Scribus on the right).
As you can probably tell, Scribus creates more condensed output. The settings were the same for both programs (automatically translated from LO styles by Scribus, not verified by hand) and LO's output file was 339 pages compared to 326 for Scribus.

Which one should you use then?

Like most things in life, that depends. If your document has a notable amount of mathematics, then you most likely want to go with LaTeX. If the document is something like a magazine or you require the highest typographical quality possible, then Scribus is a good choice. For "plain old books" the question becomes more complicated.

If you need a fully color managed workflow, then Scribus is the only viable option. If the default output of LO is good enough for you, the document has few figures and you are fine with needing to have a great battle at the end to line the images up, LO provides a fairly smooth experience.  You have to use styles properly, though, or the whole thing will end up in tears. LO is especially suitable for documents with lots of levels, headings and cross references between the two. LaTeX is also very good with those, but its unfortunate downside is that defining new styles is really hard. So is changing fonts, so you'd better be happy with Computer Modern. If the document has lots of images, then LaTeX's automatic figure floats make a ton of manual work completely disappear.

Original data

The original source documents as well as the PDF output for both programs can be found in this Github repo

Tuesday, June 8, 2021

An overhaul of Meson's WrapDB dependency management/package manager service

For several years already Meson has had a web service called WrapDB for obtaining and building dependencies automatically. The basic idea is that it takes unaltered upstream tarballs, adds Meson build definitions (if needed) as a patch on top and builds the whole thing as a Meson subproject. While it has done its job and provided many packages, the UX for adding new versions has been a bit cumbersome.

Well no more! With a lot of work from people (mostly Xavier Claessens) all of WrapDB has been overhauled to be simpler. Instead of separate repos, all wraps are now stored in a single repo, making things easier.  Adding new packages or releases now looks like this:

  • Fork the repo
  • Add the necessary files
  • Submit a PR
  • Await results of automatic CI and (non-automatic :) reviewer comments
  • Fix issues until the PR is merged
The documentation for the new system is still being written, but submissions are already open. You do need the current trunk of Meson to use the v2 WrapDB. Version 1 will remain frozen for now so old projects will keep on building. All packages and releases from v1 WrapDB have been moved to v2, except some old ones that have been replaced by something better (e.g. libjpeg has been replaced by libjpeg-turbo) so starting to use the new version should be transparent for most people.

Submitting new dependencies

Anyone can submit any dependency project that they need (assuming they are open source, of course). All you need to do is to convert the project's build definition to Meson and then submit a pull request as described above. You don't need permission from upstream to submit the project. The build files are MIT licensed so projects that want to provide native build definitions should be able to integrate WrapDB's build definitions painlessly.

Submitting your own libraries

Have you written a library that already builds with Meson and would like to make it available to all Meson users with a single command:

meson wrap install yourproject

The procedure is even simpler than above, you just need to file a pull request with the upstream info. It only takes a few minutes.

Friday, June 4, 2021

Formatting an entire book with LibreOffice, what's it like?

I have created full books using both LaTeX and Scribus. I have never done it with LibreOffice, though. The closest I've ever come was watching people write their masters' theses in Word, failing miserably and swearing profusely. To find out what it's really like, I chose to typeset an entire book from scratch using nothing else but LibreOffice.

The unfortunate thing about LibreOffice (which it inherits from MS Word compatibility) is that there is a "correct" way to use it which is the exact opposite way of how people instinctively want to use it. The proper way is to use styles for everything as opposed to applying fonts, sizes, spacing et al by hand. In practice every time I am given a Word document I check how it has been formatted. I have never seen a single document in the wild that would have gotten this right. Even something as simple as chapter indentation is almost always done with spaces.

Getting the text

Rather than using lorem ipsum or writing a book from scratch, I grabbed an existing book from Project Gutenberg. A random choice landed upon Gulliver's Travels, which turned out to be fortunate as it has several interesting and uncommon typographical choices. The source data of Project Gutenberg is UTF-8 text. All formatting has to be added afterwards. Here's what the first page ended up looking like after a few evenings' worth of work.

The source text file is line wrapped to 80 characters and chapters are separated by two or more newlines. This does not really work with LO, so the first step is to preprocess the text with a Python script so that every chapter of text is on its own (very long) line and then the text can be imported to LO. After import each sections must be assigned a proper style. The simplest approach is to select all text, apply the Text Body style and then manually seek all chapter headings and set them to Heading 1. That takes care of the formatting needs of ~95% of the text (though the remaining 5% take 10x more work).

Page layout

The original book's dimensions are not provided, so I took a random softcover book [1] from my shelf, measured it with a ruler and replicated the page settings. The book is set in the traditional style where everything up to the actual text has page numbers in roman numerals whereas the actual text uses arabic numerals. Setting it up was straightforward, though I had to create six different page styles to get the desired result.

Text layout challenges

Gulliver's Travels is a bit unusual in that every chapter begins with a small introductory text explaining what will happen in the chapter. Apparently readers in the 1720s were not afraid of spoilers.

In the Project Gutenberg source text these sections (and many others) were written in all capital letters. However it is likely that in the original book they were instead written in small caps. Fixing this would require retyping the text to be in lower case. Fortunately LO has an option in the format menu to convert text to lower case, which makes this operation fairly painless.

Another unusual thing is that the book does not have a regular table of contents, instead it duplicates these small text chapters.

LO has a builtin TOC generator but it can't handle this (I think) so the layout has to be recreated by hand with tables and manual cross reference fields. Controlling page breaks and the like is difficult and I could not make it work perfectly. The above picture has two bugs, the illustration cross reference should be in roman numerals (as it is on a preface page) but LO insists on formatting it using an arabic number. The last chapter on the left page gets split up and the page number is on the left page, whereas it should be on the right (bottom aligned). Even better would be if the chapter heading and text could be defined to always stick together and not be split over pages. There is a setting for this, but it does not seem to work inside tables

Pictures

There are several illustrations in the book and scans of the pictures were also provided by Project Gutenberg. Adding them in the document revealed that figure handling is clearly LO's weakest point (again, presumably because it inherits its model from Word). It seems that in this model each figure has an anchor point in the text and you can align the figure relative to that but the image must be on the same page as the anchor. Were it to go on the next page, LO adds a page break so that the two go to the same page. This leaves a potentially large empty space at the end of the previous page, which looks just plain weird.

In contrast this is something that LaTeX does exceptionally well with its floating figures. Basically it tries to add the figure on the current page and if it will not fit, it puts it on the next page. There does not seem to be a way to get this behaviour in LO. Or at least I could not find one, googling did not help and neither did asking for help on the lazyweb. Playing with images was also the only time I managed to crash LO, so be careful; save early, save often.

The only reasonably working solution seems to be page aligned images. This works but means that if text is edited, figures do not move along with the changes and get disconnected from their source locations. Thus image aligning must be the very last thing to be done. This approach also does not work if you are using master documents. Books with many images should probably be typeset with Scribus instead, especially if proper color management is required.

In conclusion

If you are very disciplined and use LO exactly as it should be used, the end result is actually really nice. You can, for example, change the font used for text in only one place (the base style) and the entire document gets fully reformatted, reflown and repaged in less than a second. This allows you to do invasive layout tests easily, such as finding out how much more space IBM Plex Serif takes when compared to Nimbus Roman [2]. The downside is that any cut corners will cause broken output that you can't find without manually inspecting the entire document.

IKEA effect notwithstanding laying out the text in proper form makes it a lot more enticing. The process of shaping raw text to form really makes it come alive in a sense. It would be nice if Project Gutenberg (or anyone else, really) provided properly formatted versions of their books (and in fact, some already are) because presentation really makes a difference for readability. Plain text and unformatted HTML is unfortunately quite drab to read.

[1] The Finnish edition of the first book in the Illuminatus trilogy, for the curious among you.

[2] Approximately 380 pages compared to 340.

Thursday, May 27, 2021

Managing dependencies with Meson + WrapDB

A recent blog post talked about how to build and manage dependencies with CMake and FetchContent. The example that they used was a simple GUI application using the SFML multimedia libraries and the Dear ImGui widget toolkit using the corresponding wrapper library. For comparison let's do the same with Meson.

The basics

The tl/dr version is that all you need is to install Meson, check out this repository and start the build. Meson will automatically download all dependencies as source, build them from scratch and create a single statically linked final executable. SFML's source archive contains a bunch of prebuilt dependencies for OpenAL and the like. Those are not used, everything is compiled from original sources. A total of 11 subprojects are used including things like libpng, Freetype and Flac.

They are downloaded from Meson's WrapDB dependency provider/package manager service. which combines upstream tarballs with user submitted Meson build definitions. The only exception is stb. It has no releases, instead it is expected to be used directly from Git. As WrapDB only provides actual releases, this dependency needs to be checked out from a custom Git repo. This is entirely transparent to the user, the only change is that the wrap file specifying where the dependency comes from points to a different place.

If you actually try to compile the code you might face some issues. It has only been properly tested on Windows. It will probably work on Linux and most definitely won't work on macOS. At the time of writing GNU's web mirror has an expired certificate so downloading Freetype's release tarball will fail. You can work around it by downloading it by hand and placing it in the subprojects/packagecache directory. The build of SFML might also fail as the code uses auto_ptr, which has been removed from some stdlibs. This has been fixed in master (but not in the way you might have expected) but the fix has not made it to a release yet.

What does it look like?

I would have added an inline image here, but for some reason Blogger's image uploader is broken and just fails (on two different OSs even). So here's an imgur link instead.

This picture shows the app running. To the left you can also see all the dependencies that were loaded during the build. It also tells you why people should do proper release tarballs rather than relying on Github's autogenerated ones. Since every project's files are named v1.0.0.zip, the risk of name collision is high.

What's the difference between the two?

CMake has a single flat project space (or at least that is how it is being used here) which is used like this:

FetchContent_Declare(
  sfml
  URL https://github.com/SFML/SFML/archive/refs/tags/2.5.1.zip
  URL_MD5 2c4438b3e5b2d81a6e626ecf72bf75be
)
add_subdirectory(sfml)

I.e. "download the given file, extract it in the current directory (whatever it may be) and enter it as if it was our own code". This is "easy" but problematic in that the subproject may change its parent project in interesting ways that usually lead to debugging and hair pulling.

In Meson every subproject is compiled in its own isolated sandbox. They can only communicate in specific, well defined and secured channels. This makes it easy to generate projects that can be built from source on Windows/macOS/Android/etc and which use system dependencies on Linux transparently. This equals less hassle for everyone involved.

There are other advantages as well. Meson provides a builtin option for determining whether a project should build its libraries shared or static. This option can be set on the command line per subproject. The sample application project is set up to build everything statically for convenience. However one of the dependencies, OpenAL, is LGPL, so for final distributions you'll probably need to build it as a shared library. This can be achieved with the following command:

meson configure -Dopenal-soft:default_library=shared

After this only the OpenAL dependency is built as a shared library whereas everything else is still static. As this is a builtin, no project needs to write their own options, flags and settings to select whether to build shared or static libraries. Better yet, no end user has to hunt around to discover whether the option to change is FOO_BUILD_SHARED, FOO_ENABLE_SHARED, FOO_SHARED_LIBS, SHARED_FOO, or something else.

Tuesday, May 18, 2021

Why all open source maintainers are jerks, the Drake equation hypothesis

Preface

This blog post is meant to be humorous. It is not a serious piece of scientifically rigorous research. In particular it is not aiming to justify bad behaviour or toxicity in any way, shape or form. Neither does it claim that this mechanism is the only source of negativity. If you think it is doing any of these things, then you are probably taking the whole thing too seriously and are reading into it meanings and implications that are not there. If it helps, you can think of the whole thing as part of a stand-up comedy routine.

Why is everybody a jerk?

It seems common knowledge that maintainers of major open source projects are rude. You have your linuses, lennarts, ulrichs, robs and so on. Why is that? What is it about project maintenance that brings out these supposed toxics? Why can't projects be manned by nice people? Surely that would be better.

Let's examine this mathematically. For that we need some definitions. For each project we have the following variables:

  • N, the total number of users
  • f_p, the fraction of users who would like to change the program to better match their needs
  • f_r, the fraction of users with a problem who file a change request
  • f_rej, the fraction of submitted change requests that get rejected
  • f_i, the fraction of of people who consider rejections as attacks against their person
  • f_c, the fraction of people who complain about their perceived injustices on public forums
  • f_j, the fraction of complainers formulating their complaints as malice on the maintainer's part

Thus we have the following formula for the amount of "maintainer X is a jerk" comments on the net:

J = N * f_p * f_r * f_rej * f_i * f_c * f_j

Once J reaches some threshold, the whisper network takes over and the fact that someone is a jerk becomes "common knowledge".

The only two variables that a maintainer can reasonably control are N and f_rej (note that f_i can't ever be brought to zero, because some people are incredibly good at taking everything personally) Perceived jerkness can thus only be avoided either by having no users or accepting every single change request ever filed. Neither of these is a good long term strategy.

Thursday, May 13, 2021

.C as a file extension for C++ is not portable

Some projects use .C as a file extension for C++ source code. This is ill-advised, because it is can't really be made to work automatically and reliably. Suppose we have a file source.C with the following contents:

class Foo {
public:
    int x;
};

Let's compile this with the default compiler on Linux:

$ cc -c -o /dev/null source.C

Note that that command is using the C compiler, not the C++ one. Still, the compiler will autodetect the type from the extension and compile it as C++. Now let's do the same thing using Visual Studio:

$ cl /nologo /c source.C
source.C(1): Error C2061 Syntax error: Identifier 'Foo'
<a bunch of other errors>

In this case Visual Studio has chosen to compile it as plain C. The defaults between these two compilers are the opposite and that leads to problems.

How to fix this?

The best solution is to change the file extension to an unambiguous one. The following is a simple ZSH snippet that does the renaming part:

for i in **/*.C; do git mv ${i} ${i:r}.cpp; done

Then you need to do the equivalent in your build files with search-and-replace.

If that is not possible, you need to use the /TP compiler switch with Visual Studio to make it compile the source as C++ rather than C. Note that if you use this argument on a target, then all files are built as C++, even the plain C ones. This is unreliable and can lead to weird bugs. Thus you should rename the files instead.

Wednesday, May 5, 2021

Is the space increase caused by static linking a problem?

Most recent programming languages want to link all of their dependencies statically rather than using shared libraries. This has many implications, but for now we'll only focus on one: executable size. It is generally accepted that executables created in this way are bigger than when static linking. The question is how much and whether it even mattesr. Proponents of static linking say the increase is irrelevant given current computers and gigabit networks. Opponents are of the, well, opposite opinion. Unfortunately there is very little real world measurements around for this.

Instead of arguing about hypotheticals, let's try to find some actual facts. Can we find a case where, within the last year or so, a major proponent of static linking has voluntarily switched to shared linking due to issues such as bandwidth savings. If such a case can be found, then it would indicate that, yes, the binary size increase caused by static linking is a real issue.

Android WebView, Chrome and the Trichrome library

Last year (?)  Android changed the way they provide both the Chrome browser and the System WebView app [1]. Originally both of them were fully isolated, but after the change both of them had a dependency on a new library called Trichrome, which is basically just a single shared library. According to news sites, the reasoning was this:

"Chrome is no longer used as a WebView implementation in Q+. We've moved to a new model for sharing common code between Chrome and WebView (called "Trichrome") which gives the same benefits of reduced download and install size while having fewer weird special cases and bugs."

Google has, for a long time, favored static linking. Yet, in this case, they have chosen to switch from static linking to shared linking on their flagship application on their flagship operating system. More importantly their reasons seem to be purely technical. This would indicate that shared linking does provide real world benefits compared to static linking.

[1] I originally became aware of this issue since this change broke updates on both of these apps and I had to fix it manually with apkmirror.

Tuesday, May 4, 2021

"Should we break the ABI" is the wrong question

The ongoing battle on breaking C++'s ABI seems to be gathering steam again. In a nutshell there are two sets of people in this debate. The first set wants to break ABI to gain performance and get rid of bugs, whereas the second set of people want to preserve the ABI to keep old programs working. Both sides have dug their heels in the ground and refuse to budge.

However debating whether the ABI should be broken or not is not really the issue. A more productive question would be "if we do the break, how do we keep both the old and new systems working at the same time during a transition period". That is the real issue. If you can create a good solution to this problem, then the original problem goes away because both sides get what they want. In other words, what you want to achieve is to be able to run a command like this:

prog_using_old_abi | prog_using_new_abi

and have it work transparently and reliably. It turns out that this is already possible. In fact many (most?) people are reading this blog post on a computer that already does exactly that.

Supporting multiple ABIs at the same time

On Debian-based systems this is called multi-arch support. It allows you to, for example, run 32 and 64 bit apps transparently on the same machine at the same time. IIRC it was even possible to upgrade a 32 bit OS install to 64 bits by first enabling the 64 bit arch and then disabling the 32 bit one. The basic gist of multiarch is that rather than installing libraries to /usr/lib, they get installed to something like /usr/lib/x86_64. The kernel and userspace tools know how to handle these two different binary types based on the metadata in ELF executables.

Given this we could define an entirely new processor type, let's call it x86_65, and add that as a new architecture. Since there is no existing code we can do arbitrary changes to the ABI and nothing breaks. Once that is done we can create the full toolchain, rebuild all OS packages with the new toolchain and install them. The old libraries remain and can be used to run all the old programs that could not be recompiled (for whatever reason). 

Eventually the old version can be thrown away. Things like old Steam games could still be run via something like Flatpak. Major corporations that have old programs they don't want to touch are the outstanding problem case. This just means that Red Hat and Suse can sell them extra-long term support for the old ABI userland + toolchain for an extra expensive price. This way those organizations who don't want to get with the times are the ones who end up paying for the stability and in return distro vendors get more money. This is good.

Simpler ABI tagging

Defining a whole virtual CPU just for this seems a bit heavy handed and would probably encounter a lot of resistance. It would be a lot smoother if there were a simpler way to version this ABI change. It turns out that there is. If you read the ELF specification, you will find that it has two fields for ABI versioning (and Wikipedia claims that the first one is usually left at zero). Using these fields the multiarch specification could be expanded to be a "multi-abi" spec. It would require a fair bit of work in various system components like the linker, RPM, Apt and the like to ensure the two different ABIs are never loaded in the same process. As an bonus you could do ABI breaking changes to libc at the same time (such as redefining intmax_t) There does not seem to be any immediate showstoppers, though, and the basic feasibility has already been proven by multiarch.

Obligatory Rust comparison

Rust does not have a stable ABI, in fact it is the exact opposite. Any compiler version is allowed to break the ABI in any way it wants to. This has been a constant source of friction for a long time and many people have tried to make Rust commit to some sort of a stable ABI. No-one has been successful. It is unlikely they will commit to distro-level ABI stability in the foreseeable future, possibly ever.

However if we could agree to steady and continuous ABI transitions like these every few years then that is something that they might agree to. If this happens then the end result would be beneficial to everyone involved. Paradoxically it would also mean that by having a well established incremental way to break ABI would lead to more ABI stability overall.

Sunday, April 25, 2021

The joys of creating Xcode project files

Meson's Xcode backend was originally written in 2014 or so and was never complete or even sufficiently good for daily use. The main reason for this was that at the time I got extremely frustrated with Xcode's file format and had to stop because continuing with would have lead to a burnout. It's just that unpleasant. In fact, when I was working on it originally I spontaneously said the following sentence out loud:

You know, I really wish this file was XML instead.

I did not think these words could be spoken with a straight face. But they were.

The Xcode project file is not really even a "build file format" in the sense that it would be a high level description of the build setup that a human could read, understand and modify. Instead it seems that Xcode has an internal enterprise-quality object model and the project file is just this data structure serialised out to disk in a sort-of-like-json-designed-in-1990 syntax. This format is called a plist or a property list. Apparently there is even an XML version of the format, but fortunately I've never seen one of those. Plists are at least plain text so you can read and write them, but sufficiently like a binary that you can't meaningfully diff them which must make revision control conflict resolution a joy.

The semantics of Xcode project files are not documented. The only way to really work with them is to define simple projects either with Xcode itself or with CMake, read the generated project file and try to reverse-engineer its contents from those. If you get it wrong Xcode prints a useless error message. The best you can hope for is that it prints the line number where the error occurred. Often it does not.

The essentials

An Xcode project file is essentially a single object dictionary inside a wrapper dictionary. The keys are unique IDs and the values are dictionaries, meaning the whole thing is a string–variant wrapper dictionary containing a string–variant dictionary of string–variant dictionaries. There is no static typing, each object contains an isa key which has a string telling what kind of an object it is. Everything in Xcode is defined by building these objects and then by referring to other objects via their unique ids (except when it isn't, more about this later).

Since everything has a unique ID, a reasonable expectation would be that a) it would be unique per object and b) you would use that ID to refer to the target. Neither of these are true. For example let's define a single build target, which in Xcode/CMake parlance is a PBXNativeTarget. Since plist has a native array type, the target's sources could be listed in an array of unique ids of the files. Instead the array contains a reference to a PBXSourcesBuildPhase object that has a few keys which are useless and always the same and an array of unique ids. Those do not point to file objects, as you might expect, but to a PBXBuildFile object, whose contents look like this:

24AA497CCE8B491FB93D4C76 = {
  isa = PBXBuildFile;
  fileRef = 58CFC111B9B64310B946BCE7 /* /path/to/file */;
};

There is no other data in this object. Its only reason for existing, as far as I can tell, is to point to the actual PBXFileReference object which contains the filesystem path. Thus each file actually gets two unique ids which can't be used interchangeably. But why stop at two?

In order to make the file appear in Xcode's GUI it needs more unique ids. One for the tree widget and another one it points to. Even this is not enough, though, because if the file is used in two different targets, you can not reuse the same unique ids (you can in the build definition, but not in the GUI definition just to make things more confusing). The end result being that if you have one source file that is used in two different build targets, then it gets at least four different unique id numbers.

Quoting

In some contexts Xcode does not use uids but filenames. In addition to build targets Xcode also provides PBXAggregateTargets, which are used to run custom build steps like code generation. The steps are defined in a PBXShellScriptBuildPhase object whose output array definition looks like this:

outputPaths = (
    /path/to/output/file,
);

Yep, those are good old filesystem paths. Even better, they are defined as an actual honest to $deity array rather than a space separated string. This is awesome! Surely it means that Xcode will properly quote all these file names when it invokes external commands.

Lol, no!

If your file names have special characters in them (like, say, all of Meson's test suite does by design) then you get to quote them manually. Simply adding double quotes is not enough, since they are swallowed by the plist parser. Instead you need to add additional quoted quote characters like this: "\"foo bar\"".  Seems simple, but what if you need to pass a backslash through the system, like if you want to define some string as "foo\bar"? The common wisdom is "don't do that" but this is a luxury we don't have, because people will expect it to work, do it anyway and report bugs when it fails.

To cut a long and frustrating debugging session short, the solution is that you need to replace every backslash with eight backslashes and then it will work. This implies that the string is interpreted by a shell-like thing three times. I could decipher where two of them occur but the third one remains a mystery. Any other number of backslashes does not work and only leads to incomprehensible error messages.

Quadratic slowdowns for great enjoyment

Fast iterations are one of the main ingredients of an enjoyable development experience. Unfortunately Xcode does not provide that for this use case. It is actually agonizingly slow. The basic Meson test suite consists of around 240 sample projects. When using the Ninja backend it takes 10 minutes to configure, build and run the tests for all of them. The Xcode backend takes 24 minutes to do the same. Looking at the console when xcodebuild starts it first reads its input files, then prints "planning build", pauses for a while and then starts working. This freeze seems to take longer than Meson took configuring and generating the project file. Xcodebuild lags even for projects that have only one build target and one source file. It makes you wonder what it is doing and how it is even possible to spend almost a full second planning the build of one file. It also makes you wonder how a pure Python program written by random people in their spare time outperforms the flagship development application created by the richest corporation in the world.

Granted, this is a very uncommon case as very few people need to regenerate their project files all the time. Still, this slowness makes developing an Xcode backend a tedious job. Every time you add functionality to fix one test case, you have to rerun the entire test suite. This means that the further along you get, the slower development becomes. By the end I was spending more time on Youtube waiting for tests to finish than I did writing code.

Final wild speculation

Xcode has a version compatibility selection box that looks like this:

This is extremely un-Apple-like. Backwards compatibility is not a thing Apple has ever really done. Typically you get support for the current version and, if you are really lucky, the previous one. Yet Xcode has support for versions going all the way back to 2008. In fact, this might be the product with the longest backwards compatibility story ever provided by Apple. Why is this?

We don't really know. However being the maintainer of a build system means that sometimes people tell you things. Those things may be false or fabricated, of course, so the following is just speculation. I certainly have no proof for any of it. Anyhow it seems that a lot the fundamental code that is used to build macOS exists only in Xcode projects created ages ago. The authors of said projects have since left the company and nobody touches those projects for fear of breaking things. If true this would indicate that the Xcode development team has to keep those old versions working. No matter what.

Sunday, April 11, 2021

Converting a project to Meson: the olc Pixel Game Engine

Meson's development has always been fairly practical focusing on solving real world problems people have. One simple way of doing that is taking existing projects, converting their build systems to Meson and seeing how long it takes, what pain points there are and whether there are missing features. Let's look at one such conversion.

We'll use the One Lone Coder Pixel Game Engine. It is a simple but fairly well featured game engine that is especially suitable for beginners. It also has very informative YouTube channel teaching people how to write their own computer games. The engine is implemented as a single C++ header and the idea is that you can just copy it in your own project, #include it and start coding your game. Even though the basic idea is simple, there are still some build challenges:

  • On Windows there are no dependencies, it uses only builtin OS functionality but you need to set up a Visual Studio project (as that is what most beginners tend to use)
  • On Linux you need to use system dependencies for the X server, OpenGL and libpng
  • On macOS you need to use both OpenGL and GLUT for the GUI and in addition obtain libpng either via a prebuilt library or by building it yourself from source

The Meson setup

The Meson build definition that provides all of the above is 25 lines long. We'll only look at the basic setup, but the whole file is available in this repo for those who want all the details. The most important bit is looking up dependencies, which looks like this:

external_deps = [dependency('threads'), dependency('gl')]
cpp = meson.get_compiler('cpp')

if host_machine.system() == 'windows'
  # no platform specific deps are needed
elif host_machine.system() == 'darwin'
  external_deps += [dependency('libpng'),
                    dependency('appleframeworks', modules: ['GLUT'])]
else
  external_deps += [dependency('libpng'),
                    dependency('x11'),
                    cpp.find_library('stdc++fs', required: false)]

The first two dependencies are the same on all platforms and use Meson's builtin functionality for enabling threading support and OpenGL. After that we add platform specific dependencies as described above. The stdc++fs one is only needed if you want to build on Debian stable or Raspberry Pi OS, as their C++ standard library is old and does not have the filesystem parts enabled by default. If you only support new OS versions, then that dependency can be removed.

The interesting bit here is libpng. As macOS does not provide it as part of the operating system, we need to build it ourselves. This can be accomplished easily by using Meson's WrapDB service for package management. Adding this dependency is a matter of going to your project's source root, ensuring that a directory called subprojects exists and running the following command:

meson wrap install libpng

This creates a wrap file that Meson can use to download and build the dependency as needed. That's all that is needed. Now anyone can fork the repo, edit the sample program source file and get going writing their own game.

Bonus Xcode support

Unlike the Ninja and Visual Studio backends, the Xcode backend has always been a bit ...  crappy, to be honest. Recently I have started work on bringing it up to feature parity with the other backends. There is still a lot of work to be done, but it is now good enough that you can build and debug applications on macOS. Here is an example of the Pixel engine sample application running under the debugger in Xcode.


This functionality will be available in the next release (no guarantees on feature completeness) but the impatient among you can try it out using Meson's Git trunk.

Wednesday, March 31, 2021

Never use environment variables for configuration

Suppose you need to create a function for adding two numbers together in plain C. How would you write it? What sort of an API would it have? One possible implementation would be this:

int add_numbers(int one, int two) {
    return one + two;
}

// to call it you'd do
int three = add_numbers(1, 2);

Seems reasonable? But what if it was implemented like this instead:

int first_argument;
int second_argument;

void add_numbers(void) {
    return first_argument + second_argument;
}

// to call it you'd do
first_argument = 1;
second_argument = 2;
int three = add_numbers();

This is, I trust you all agree, terrible. This approach is plain wrong, against all accepted coding practices and would get immediately rejected in any code review. It is left as an exercise to the reader to come up with ways in which this architecture is broken. You don't even need to look into thread safety to find correctness bugs.

And yet we have environment variables

Environment variables is exactly this: mutable global state. Envvars have some legitimate usages (such as enabling debug logging) but they should never, ever be used for configuring core functionality of programs. Sadly they are used for this purpose a lot and there are some people who think that this is a good thing. This causes no end of headaches due to weird corner, edge and even common cases.

Persistance of state

For example suppose you run a command line program that has some sort of a persistent state.

$ SOME_ENVVAR=... some_command <args>

Then some time after that you run it again:

$ some_command <args>

The environment is now different. What should the program do? Use the old configuration that had the env var set or the new one where it is not set? Error out? Try to silently merge the different options into one? Something else?

The answer is that you, the end user, can not now. Every program is free to do its own thing and most do. If you have ever spent ages wondering why the exact same commands work when run from one terminal but not the other, this is probably why.

Lack of higher order primitives

An environment variable can only contain a single null-terminated stream of bytes. This is very limiting. At the very least you'd want to have arrays, but it is not supported. Surely that is not a problem, you say, you can always do in-band signaling. For example the PATH environment variable has many directories which are separated by the : character. What could be simpler? Many things, it turns out.

First of all the separator for paths is not always :. On Windows it is ;. More generally every program is free to choose its own. A common choice is space:

CFLAGS='-Dfoo="bar" -Dbaz' <command>

Except what if you need to pass a space character as part of the argument? Depending on the actual program, shell and the phase of the moon, you might need to do this:

ARG='-Dfoo="bar bar" -Dbaz'

or this:

ARG='-Dfoo="bar\ bar" -Dbaz'

or even this:

ARG='-Dfoo="bar\\ bar" -Dbaz'

There is no way to know which one of these is the correct form. You have to try them all and see which one works. Sometimes, as an implementation detail, the string gets expanded multiple times so you get to quote quote characters. Insert your favourite picture of Xzibit here.

For comparison using JSON configuration files this entire class os problems would not exist. Every application would read the data in the same way, because JSON provides primitives to express these higher level constructs. In contrast every time an environment variable needs to carry more information than a single untyped string, the programmer gets to create a new ad hoc data marshaling scheme and if there's one thing that guarantees usability it's reinventing the square wheel.

There is a second, more insidious part to this. If a decision is made to configure something via an environment variable then the entire design goal changes. Instead of coming up with a syntax that is as good as possible for the given problem, instead the goal is to produce syntax that is easy to use when typing commands on the terminal. This reduces work in the immediate short term but increases it in the medium to long term.

Why are environment variables still used?

It's the same old trifecta of why things are bad and broken:

  1. Envvars are easy to add
  2. There are existing processes that only work via envvars
  3. "This is the way we have always done it so it must be correct!"
The first explains why even new programs add configuration options via envvars (no need to add code to the command line parser, so that's a net win right?).

The second makes it seem like envvars are a normal and reasonable thing as they are so widespread.

The third makes it all but impossible to improve things on a larger scale. Now, granted, fixing these issues would be a lot of work and the transition would unearth a lot of bugs but the end result would be more readable and reliable.

Monday, March 22, 2021

Writing a library and then using it as a Meson dependency directly from upstream Git

Meson has many ways of obtaining dependencies. The most common is pkg-config for prebuilt dependencies and the WrapDB for building upstream releases from source. A lesser known way is that you can get dependencies [1] directly from the upstream project's Git repository. Meson will transparently download and build them for you. There does not seem to be that many examples of this on the Internet, so let's see how one would both create and consume dependencies in this way.

The library

Rather than creating a throwaway library, let's instead make one that is actually useful. The C++ standard library does not have a full featured way to split strings, meaning that every project needs to write their own. To simplify the design, we're going to steal shamelessly from Python. That is, when in doubt, try to behave as closely as possible to how Python's string splitter functions work. In addition:

  • support any data storage (i.e. all input parameters should be string views)
  • have helper functionality for mmap, so even huge files can be split without massive overhead
  • return types can be either efficient (string views to the input) or safe (strings with copied data)

Once you start looking into the implementation it very quickly becomes clear why this functionality is not already in the standard library. It is quite tricky and there are many interesting things in Python's implementation that most people have never noticed. For example splitting a string via whitespace does this:

>>> ' hello world '.split()
['hello', 'world']

which is what you'd expect. But note that the only whitespace characters are spaces. So what happens if we optimise the code and explicitly split only by space?

>>> ' hello world '.split(' ')
['', 'hello', 'world', '']

That's ... unexpected. It turns out that if you split by whitespace, Python silently removes empty substrings. You can't make it not do that if you specify your own split criterion. This seems like a thing a general solution should provide.

Another common idiom in Python is to iterate over lines in a file with this:

for line in open('infile.txt'):
    ...

This seems like a thing that could be implemented by splitting the file contents with newline characters. That works for files whose path separator is \n but fails with DOS line endings of \ŗ\n. Usually in string splitting the order of the input characters does not matter, but in this case it does. \r\n is a single logical newline, whereas \n\r is two [2]. Further, in Python the returned strings contain the line ending characters converted to \n, but in this is not something we can do. Opening a DOS file should return a string view to the original immutable data but the \r character should be a \n instead. This could only be done by returning a modified copy rather than a view to the original data. This necessitates a behavioural difference to Python so that the linefeed characters are omitted.

This is the kind of problem that would be naturally implemented with coroutines. Unfortunately those are c++20 only, so very few people could use it and there is not that much info online on how to write your own generators. So vectors of string_views it is for now.

The implementation

The code for the library is available in this Github repo. For the purposes of this blog post, the interesting line is this one specifying the dependency information:

psplit_dep = declare_dependency(include_directories: '.')

This is the standard way subprojects set themselves up to be used. As this is a header only library, the dependency only has an include directory.

Using it

A separate project that uses the dependency to implement the world's most bare bones CSV parser can be obtained here. The actual magic happens in the file subprojects/psplit.wrap, which looks like this:

[wrap-git]
directory = psplit
url = https://github.com/jpakkane/psplit.git
revision = head

[provide]
psplit = psplit_dep

The first section describes where the dependency can be downloaded and where it should be placed. The second section specifies that this repository provides one dependency named psplit and that its dependency information can be found in the subproject in a variable named psplit_dep.

Using it is simple:

psplit_dep = dependency('psplit')
executable('csvsplit', 'csvsplit.cpp',
    dependencies: psplit_dep)

When the main project requests the psplit dependency, Meson will try to find it, notices that a subproject does provide it and will then download, configure and build the dependency automatically.

Language support

Even though we used C++ here, this works for any language supported by Meson. It even works for mixed language projects, so you can for example have a library in plain C and create bindings to it in a different language.

[1] As long as they build with Meson.

[2] Unless you are using a BBC Micro, though I suspect you won't have a C++17 compiler at your disposal in that case.

Friday, March 19, 2021

Microsoft is shipping a product built with Meson

Some time ago Microsoft announced a compatibility pack to get OpenGL and OpenCL running even on computers whose hardware does not provide native OpenGL drivers. It is basically OpenGL-over-Direct3D. Or that is at least my understanding of it, hopefully this description is sufficiently accurate to not cause audible groans on the devs who actually know what it is doing under the covers. More actual details can be found in this blog post.

An OpenGL implementation is a whole lot of work and writing one from scratch is a multi-year project. Instead of doing that, Microsoft chose the sensible approach of taking the Mesa implementation and porting it to work on Windows. Typically large corporations do this by the vendoring approach, that is, copying the source code inside their own repos, rewriting the build system and treating it as if it was their own code.

The blog post does not say it, but in this case that approach was not taken. Instead all work was done in upstream Mesa and the end products are built with the same Meson build files [1]. This also goes for the final release that is available in Windows Store. This is a fairly big milestone for the Meson project as it is now provably mature enough that major players like Microsoft are willing to use it to build and ship end user products. 

[1] There may, of course, be some internal patches we don't know about.

Wednesday, March 10, 2021

Mixing Rust into an existing C shared library using Meson

Many people are interested in adding Rust to their existing projects for additional safety. For example it would be convenient to use Rust for individual high-risk things like string parsing while leaving the other bits as they are. For shared libraries you'd need to be able to do this while preserving the external plain C API and ABI. Most Rust compilation is done with Cargo, but it is not particularly suited to this task due to two things.

  1. Integrating Cargo into an existing project's build system is painful, because Cargo wants to dominate the entire build process. It does not cooperate with these kind of build setups particularly well.
  2. Using any Cargo dependency brings in tens or hundreds of dependency crates including five different command line parsers, test frameworks and other deps that you don't care about and don't need but which take forever to compile.
It should be noted that the latter is not strictly Cargo's fault. It is possible to use it standalone without external deps. However what seems to happen in practice is all Cargo projects experience a dependency explosion sooner or later. Thus it would seem like there should be a less invasive way to merge Rust into an existing code base. Fortunately with Meson, there is.

The sample project

To see how this can be done, we created a simple standalone C project for adding numbers. The full source code can be found in this repository. The library consists of three functions:

adder* adder_create(int number);
int adder_add(adder *a, int number);
void adder_destroy(adder*);

To add the numbers 2 and 4 together, you'd do this:

adder *two_adder = adder_create(2);
int six = adder_add(two_adder, 4);
adder_destroy(two_adder);

As adding numbers is highly dangerous, we want to implement the adder_add function in Rust and leave the other functions untouched. The implementation in all its simplicity is the following:

#[repr(C)]
pub struct Adder {
  pub number: i32
}

#[no_mangle]
pub extern fn adder_add(a: &Adder, number: i32) -> i32 {
    return a.number + number;
}

The build setup

Meson has native support for building Rust. It does not require Cargo or any other tool, it invokes rustc directly. In this particular case we need to build the Rust code as a staticlib.

rl = static_library('radder', 'adder.rs',
                    rust_crate_type: 'staticlib')

In theory all you'd need to do, then, is to link this library into the main shared library, remove the adder_add implementation from the C side and you'd be done. Unfortunately it's not that simple. Because nothing in the existing code calls this function, the linker will look at it, see that it is unused and throw it away.

The common approach in these cases is to use link_whole instead of plain linking. This does not work, because rustc adds its own metadata files inside the static library. The system linker does not know how to handle those and will exit with an error. Fortunately there is a way to make this work. You can specify additional undefined symbol names to the linker. This makes it behave as if something in the existing code had called adder_add, and grabs the implementation from the static library. This can be done with an additional kwarg to the shared_library call.

link_args: '-Wl,-u,adder_add'

With this the goal has been reached: one function implementation is done with Rust while preserving both the API and the ABI and the test suite passes as well. The resulting shared library file is about 1 kilobyte bigger than the plain C one (though if you build without optimizations enabled, it is a whopping 14 megabytes bigger).

Wednesday, February 24, 2021

Millennium prize problems but for Linux

There is a longstanding tradition in mathematics to create a list of hard unsolved problems to drive people to work on solving them. Examples include Hilbert's problems and the Millennium Prize problems. Wouldn't it be nice if we had the same for Linux? A bunch of hard problems with sexy names that would drive development forward? Sadly there is no easy source for tens of millions of euros in prize money, not to mention it would be very hard to distribute as this work would, by necessity, be spread over a large group of people.

Thus it seems is unlikely for this to work in practice, but that does not prevent us from stealing a different trick from mathematicians' toolbox and ponder how it would work in theory. In this case the list of problems will probably never exist, but let's assume that it does. What would it contain if it did exist? Here's one example I came up with. it is left as an exercise to the reader to work out what prompted me to write this post.

The Memory Depletion Smoothness Property

When running the following sequence of steps:
  1. Check out the full source code for LLVM + Clang
  2. Configure it to compile Clang and Clang-tools-extra, use the Ninja backend and RelWithDebInfo build type, leave all other settings to their default values
  3. Start watching a video file with VLC or a browser
  4. Start compilation by running nice -19 ninja
The outcome must be that the video playback works without skipping a single frame or audio sample.

What happens currently?

When Clang starts linking, each linker process takes up to 10 gigabytes of ram. This leads to memory exhaustion, flushing active memory to swap and eventually crashing the linker processes. Before that happens, however, every other app freezes completely and the entire desktop remains frozen until things get swapped back in to memory, which can take tens of seconds. Interestingly all browser tabs are killed before linker processes start failing. This happens both with Firefox and Chromium.

What should happen instead?

The system handles the problematic case in a better way. The linker processes will still die as there is not enough memory to run them all but the UI should never noticeably freeze. For extra points the same should happen even if you run Ninja without nice.

The wrong solution

A knee-jerk reaction many people have is something along the lines of "you can solve this by limiting the number of linker processes by doing X". That is not the answer. It solves the symptoms but not the underlying cause, which is that bad input causes the scheduler to do the wrong thing. There are many other ways of triggering the same issue, for example by copying large files around. A proper solution would fix all of those in one go.