Please provide tarball releases of your projects

A recent trend in open source projects seems to be to avoid releasing proper release archives (whether signed with GPG or not). Instead people add Git tags for release commits and call it a day.

A long and arduous debate could be had whether this is "wrong", "right" and whether Git hashes are equivalent to proper tarballs or not, or if --depth=1 is a good thing or not. We're not going to get into that at all.

Instead I'd like to kindly ask that all projects that do releases of any kind to provide actual release tarballs for the following two reasons:

1. It takes very little effort on your part.
2. Proper release archives make things easier for many people consuming your project.

This makes sense just from a pure numbers game perspective: a little work on your part saves a lot of work for many other people. So please do it.

What about Github automatic archive generation?

Github does provide links to download any project commit (and thus release) as an archive. This is basically a good thing but it has two major issues.

1. The filenames are of type v1.0.0.tar.gz. So from a file name you can't tell what it contains and further if you have two dependencies with the same version number, the archive files will have the same name and thus clash. Murphy's law says that this is inevitable.
2. The archives that Github generates are not stable. Thus if you redownload them the files may have different checksums. This is bad because doing supply chain verification by comparing hashes will give you random failures.

The latter might get fixed if Github changes their policy to guaranteed reproducible downloads. The former problem still exists.

A simple webui-only way to do it

If you don't want to use git archive to generate your releases for whatever reason, there is a straightforward way of doing the release using only the web ui.

1. Create your release by tagging as usual.
2. Download the Github autogenerated tarball with a browser (it does not matter whether you choose zip or tar as the format, either one is fine).
3. Rename the v1.0.0.tar.gz file to something like myproject-1.0.0.tar.gz.
4. Go to the project tags page, click on "create a new release from tag".
5. Upload the file from step #3 as a release file.

Supporting external modules in Godot game engine with Meson

The disclaimer

None of this is in upstream Godot yet. It is only a proposal. The actual code can be obtained from the meson2 branch of this repository. Further discussion on the issue should be posted here.

The problem

Godot's code base is split into independent modules that can be enabled and disabled at will. However many games require custom native code and thus need to define their own modules. The simplest way to do this (which, I'm told, game developers quite often do) is to fork the upstream repo and put your your code in it. This works and is a good solution for one-off projects that write all extra code by themselves. This approach also has its downsides. The two major ones are that updating to a newer version of the engine can turn into a rebasing hell and that it is difficult to combine multiple third party modules.

Ideally what you'd want to do is to take upstream Godot and then take a third party module for, say, physics and a second module written by completely different people that does sound effects processing, combine all three and have things just work. Typically those modules are developed in their own repositories. Thus we'd end up with the following kind of a dependency graph.

This would indicate a circular dependency between the two repositories: in order to build the external module you need to depend on the Godot repo and in order to build the Godot you need to depend on the external repository. This is bad. If there's one thing you don't want in your source code it is circular dependencies.

Solving it using Meson's primitives

If you look at the picture in more detail you can tell that there is no circular dependencies between individual targets. Thus to solve the problem you need some way to tell the external dependency how to get the core libraries and headers it needs and conversely a way for the main build to extract from the external project what modules it has built and where they are. As Meson subprojects are built in isolation one can't just blindly poke the innards of other projects as one can do if everything is in a single megaproject.

The way Godot's current build is set up is that first it defines the core libraries, then all the modules and finally things on top of that (like the editor and main executable). We need to extend this so that external modules are set up at the same time as internal modules and then joined into one. Thus the final link won't even be able to tell the difference between external and internal modules.

First we need to set up the dependency info for the core libraries, which is done like this:

godotcore_dep = declare_dependency(include_directories: INCDIRS,
                                   compile_args: CPP_ARGS,
                                   link_args: LINK_ARGS)

meson.override_dependency('godotcore', godotcore_dep)

First we set up a dependency object that encapsulates everything needed to build extension modules and then specify that whenever a dependency called godotcore is looked up, Meson will return the newly defined object. This even works inside subprojects that are otherwise isolated from the master project.

Assuming we have a list of external module subprojects available, we can go through them one by one and build them.

foreach extmod : external_modules

    sp = subproject(extmod)

    MODULE_DEPENDENCIES += sp.get_variable('module_dep')

    MODULES_ENABLED += sp.get_variable('module_names')

endforeach

The first line runs the subproject, the latter two are ignored for now, we'll come back to them. The subproject's meson.build file starts by getting the dependency info.

godotcore_dep = dependency('godotcore')

Then it does whatever it needs to build the extension module. Finally it defines the information that the main Godot application needs to use the module:

module_dep = declare_dependency(link_with: lib_module_tga,

  include_directories: '.')

module_names = ['tga']

In this case I have converted Godot's internal tga module to build as an external module hence the name. This concludes the subproject and execution resumes in the master project and the two remaining lines grab the module build information and name and append them to the list of modules to use.

This is basically it. There are obviously more details needed like integrating with Godot's documentation system for modules but the basic principle for those is the same. With this approach the integration of multiple external modules is simple: you need to place them in the main project's subprojects directory and add their names to the list of external module subprojects. All of this can be done with no code changes to the main Godot repo so updating it to the newest upstream version is just a matter of doing a git pull.

Typesetting an Entire Book Part IV: The Content

In previous blog posts (such as seals this one) we looked into typesetting a book with various FOSS tools. Those have used existing content from Project Gutenberg. However it would be a whole lot nicer to do this with your own content, especially since a pandemic quarantine has traditionally been a fruitful time to write books. Thus for completeness I ventured out to write my own. After a fair bit of time typing, retyping, typesetting, imposing, printing, gluing, sandpapering and the like, here is the 244 page product that eventually emerged from the pipeline.

As you can probably tell, this was the first time i did gouache lettering with a brush. It behaved differently from what I expected and due to reasons I could only do this after the cover had been attached to the text block, so there was no going back or doing it over.

What's its name in English and what genre does it represent?

The first one of these is actually quite a difficult question. I spent a lot of time trying to come up with a working translation but failed. Not only is the title a pun, it contains an intentional spelling error. A literal translation would be something like The First Transmission Giltch, though that misses the fact that the original name contains the phrase First Contact. The working title for the book was Office Space in Space.

As you can probably tell, the book is a sci-fi satire about humanity's first contact with alien civilisations, but it can be seen as an allegory of a software startup company. ISO standardisation also plays a minor part, and so do giant doughnuts, unexpected gravities, a cosplay horse and even space sex (obviously).

At this point many of you have probably asked the obvious question, namely isn't this just a blatant Hitchhiker's Guide to the Galaxy ripoff?

Not really. Or at least I'd like to think that it isn't. This can be explained with an analogy: what Hitchhiker's is to Star Wars, this book is to Star Trek. More specifically it is not high fantasy, but more technical, down to earth and gritty, for lack of a better term.

Can I read it?

You almost certainly can not.

In fact, let's be scientific and estimate how unlikely it would be. The first hurdle is getting the book published. Statistics say that only one book out of a thousand offered to publishers actually gets published. Even if it did get published and you had a physical copy in your hands, you probably still could not read it, since it is written in Finnish, a language that is understood only by 0.1 percent of the planet's population. If we estimate how many people who could read it actually would read it then the chances are again roughly one of a thousand.

Putting all these together and assuming a planetary population of 7 billion we find that the book will only be read by around seven people. Thus far I have convinced five of my friends to read the preview version so there are only two slots available. Your chances of being one of the two are thus quite slim. On the other hand if you are an extreme hipster, the kind who likes to boast to their friends that they only read books in languages they don't speak and which require their readers to understand both French and Latin in order to get some of the jokes within, you may have found your Citizen Kane.

If you can find it, that is.

Converting Godot game engine to Meson, how you can help

There has long been interest in switching the Godot game engine from its current SCons build system to Meson. The actual details are here, but as a quick summary:

• The original port was written by community members.
• It got stale, so I have been keeping it rebased against current master
• The original plan was to review and possibly merge it immediately after the 4.0 alpha release.

Unfortunately the people who should do the review are currently very busy with other things and can't spend much time on this. It would be important to get the change in before the 4.0 release as changing build systems mid-release is apparently a difficult task.

How can you participate?

As the actual devs are busy, what we can do is try to make things as easy for them as possible. In other words the port needs review and testing. The simplest way is to help is to check out the code from the meson branch of this repo, compiling it and running it. Then report your success (or failure) in this Github discussion thread. I will keep rebasing the branch against upstream master at regular intervals.

Note that if you encounter bugs, please do not file them against the upstream project unless you have verified that they also occur with a regular SCons build.

Currently it compiles and runs for me on Linux, Windows and macOS, but the more people can verify it the better.

It would be especially useful if people could test it on Android. FWICT the original uses Android Studio for the Java bits and somehow uses SCons to build a shared library that is used. The Meson build does build the shared lib, but it has not been tested. If someone with Android dev experience could set up and test the whole pipeline it would be great.

The same goes for iOS, though I know even less how it should be set up as I have not really done iOS development. It should build for iOS (there is a cross file in the repo) but FWICT it has never been tested beyond that.

The eventual goal

The aim is to make the Meson port as polished and tested as possible so that once upstream makes their final decision it should be as easy as possible for them (regardless of what they eventually end up choosing). The more people have tried it, the more confidence there is that the port can actually do all the things that are needed.

Compiling LibreOffice with Meson even further

After building the basics of LO on Windows and macOS the obvious next step is to build all of it. This is just grunt work and quite boring at that actually. I almost got it done apart from the fact that at the end I got lazy and skipped those bits that require weird dependencies (of which more later).

The end result has approximately 7800 individual compilation and linking steps. It takes about 30 minutes on a 16 core Ryzen 3700X CPU using Windows and Visual Studio. On a 7 year old 4 core Macbook it takes around 3 hours.

What is still needed to make it work?

Quite a bit, actually. The most pertinent would be all the configuration files that get installed. There are a lot of them and they need to be exactly right, otherwise the end result fails to start with cryptic error messages, if any. Unlike code compilation there is no easy way to know in advance what should be done or how things should behave. Ideally you'd get help with people who know the innards of the program and all the configgen bits.

The next bit is Java. You can compile LO without Java at all and, from what I can tell, there are plans to replace all Java with native code. Unfortunately that will take some time so in the mean time something would need to be done with this. Meson does have native Java support but, as far as I know, it has never been tested with huge projects so there would probably be some teething problems.

There are also many parts of existing native code that has not been ported due to missing dependencies. There many plugins for different SQL server connectors. They all need the respective client libs. These have not been ported.

All of that would handle perhaps 80% of the work. The remaining 80% would be split evenly across a bunch of places from deployment scripts to CI config to workspace changes and so on. There is arguably a third 80% bit, namely convincing people to adopt the new system.

What horrors interesting features were found lurking in the dependencies?

Oh boy, were there some. In the interest of fairness their names have been anonymized for obvious reasons.

One of the dependencies has not had a release in six years. It builds only with Autotools, generates a config.h header and all that. During porting it seemed that sometimes changes to the configuration header are not picked up. After enough digging it turned out that the header is not included anywhere. It is not even force included with a compiler flag. It is 100% useless.

This is problematic because spawning external processes to do all SIZEOF_INT check and the like are by far the slowest part of Meson and not something that can be easily sped up. Even worse, 99% of the time those checks would still be useless today. If you need integers of a specific size, use stdint.h instead. If your library has its own named integer types, define those via entries provided by stdint.h instead of hacking things by hand. It is faster, more reliable and less effort. It even works on Visual Studio since several years ago. There are still reasons to do the manual thing, but they are extremely rare. So unless you are doing something very low level like GLib or have to support Ultrix from 1992 then there really is no reason not to use stdint.h.

Another dependency claims to support Windows but it does not have any symbol visibility exports but instead relies on GCC exporting all symbols by default so it can't be build as a shared library using Visual Studio, only MinGW.

While Autotools is peculiar in many ways, one of the nicest things about it is that the config.h header it creates has a very rigid form. Thus it is a lot easier to create a tool that reads that file and works out backwards what checks it has done. Meson has a tool for this and converting Autoconf checks with it is actually quite simple. Sadly sometimes people feel the need to reinvent their own. One of LO's dependencies does exactly that. Its config header template has the regular defines. It also has lines like this:

${FOOBAR__O_SOMETHINGSOMETHING}

After a lot of inscrutable macro code written in a Certain Make-based system splattered about the source tree, the end result eventually gets expanded to this:

#define internalname func_name_in_current_libc

It is unclear why this was done and what the advantage was, but the downside is clear. There is no automatic way to convert this, it can only be reverse engineered and reimplemented by a human being. This is slow.

In general there are a lot of weird checks in various projects all of which take time to do. For example there is a fairly modernish C++ dependency that checks for the sleep C function rather than using the builtin this_thread::sleep_for function that has existed for over 11 years now. There are also checks for whether the printf function is available (because, you know, some of your users might be living in the year 1974, one can never be sure about these things). There is even a case where this test fails on the Visual Studio compiler.

All of this is to say that I have now gone with this about as far as I'm willing to do on my own. If there are people who want to see this conversion properly happen, now would be the time to get involved.

Porting the LO Windows build to macOS

In the last blog post we looked at compiling a subsection of LibreOffice on Windows using nothing but Meson and WrapDB. Now that it is working (somewhat) the obvious followup question is how much work would it be to make that build on macOS as well.

Not all that much, it turns out. The work consisted mostly of adding some defines, adding platform-specific source files and the like. It took me less than a day. A sizable fraction of that was spent waiting for my old 4-core laptop to finish compiling the code.

Building a part of LibreOffice on Windows using only Meson and WrapDB

In earlier posts (starting from this one) I ported LibreOffice's build system to Meson. The aim has not been to be complete, but to compile and link the main executables. On Linux this is fairly easy as you can use the package manager to install all dependencies (and there are quite a few of them).

One of the goals of WrapDB has been to provide external dependencies automatically on platforms lacking system dependencies (and even on Linux if you need newer dependency versions than your distro provides). It is already being used by many people and from what I've been told it works fairly well. When it comes to bigger projects like LO, there have been two major opposing view:

1. That just can't work.
2. Even if it did work, converting all dependencies would be way too much work so it could never be done.

There is only one way to counter opinions such as these and that is to do the actual work. So I set out to build LO Writer and all its dependencies using nothing but Visual Studio, Meson and WrapDB. There is to be no MinGW, Msys, Cygwin or any other unix compatibility layer.

The porting work was straightforward. Start compilation, wait for an error, typically due to missing dependencies. Then port that to Meson, submit it to WrapDB and continue.

Did it succeed?

Yes. In a way. Here's a screen shot of the extracted subprojects that were downloaded via WrapDB.

Does it run?

Lol no. It did not even run on Linux properly, because LO requires a ton of configuration files to be installed "just right" in order to start and that part had never been compiled.

Does it compile?

It does on my machine. It probably won't do so on yours. Some of the deps I used could not be added to WrapDB yet or are missing review. If you want to try, the code is here.

The problematic (from a build system point of view) part of compiling an executable and then running it to generate source code for a different target works without problems. In theory you should be able to generate VS project files and build it with those, but I only used Ninja because it is much faster.

What was hard?

The nemesis of any porting effort of LO is the i18npool subdirectory. It builds programs that convert hyphenation rules from XML files to code. It uses the ICU library for that. The basic problem of Windows is that there is no concept of RPATH (unless you fake it in) so if your binaries use shared libraries then you can't just run them. Fortunately Meson handles this transparently by wrapping binary invocations and does all the needed PATH magling needed to make things work.

However ICU's hyphenation programs are special. They also need to access some data files. On a system-wide install they are read from the common directories, but they are not available when building yourself. There are command line options to point the programs to the proper place but at the time I got frustrated and just copied the pregenerated source file from a Linux build and called it a day.

I had to do the same thing for the outputs of Flex, Bison and gperf for similar reasons. These are all fixable, but some of the generator bits also use cthulhuan shell pipelines to do "stuff". These would need to be converted to Python for portability (and also readability).

Boost

LO uses a lot of Boost. I suspected this to be a problem but fortunately it did not. Most code uses the header-only parts so those all get set by a single declare_dependency. There were a couple of uses of libraries that require actual code. One of them was for Boost Filesystem. Assuming the code does not do anything weird, that could probably be fixed to use std::filesystem instead.

The Boost code is copied in the LO repo for now. It is not added to WrapDB yet as it is quite incomplete and only builds for this use case. Still, Boost is a popular dependency so maybe having it in WrapDB would be useful, even in an incomplete state.

Could a full port be made?

Let's say that thus far there has been nothing to indicate that it would not work. The downside is that it would be a fair amount of work and it is not the cool kind where you get to write new features but instead it is the equivalent of ditch digging. Even more problematically it probably could not be done by "one person on their own" but would instead require buy-in and cooperation from a large group of developers. As people are perennially busy, getting the necessary resources would probably be challenging.

All of that being said, there is a GSoC project for doing a porting experiment. So if you are the sort of person who won't shy away from a challenge, you might consider applying.

Bonus question

How many XML parsers does LO have?

The first one is libxml. The second one is Expat. The third one is Boost's Property tree, which has its own parser (according to the docs at least, dunno if it is used in this code). The fourth one is the bunch of Awk regexps that are used in the build scripts buried inside Makefiles.

There may be more.

Portability is not sufficient for portability

A forenote

This blog post has some examples of questionable quality. This should not be meant as an attack on those projects. The issues listed here are fairly widespread, these are just the examples I ran into while doing other work.

What is meant by portability?

Before looking into portable software, let's first examine portability from a hardware perspective. When you ask most people what they consider a "portable computer", they'll probably think of laptops or possibly even a modern smartphone. But what about this:

This is most definitely a computer (the one I'm using to write this blog post, in fact), but not portable. It weighs something on the order of 10 kilos and it is too big to comfortably wrap your hands around.

And yet, I have carried this computer from one end of the Helsinki metropolitan region to another. It took over an hour on a train and a subway. When I finally got it home my arms were so exhausted that for a while I could not even lift them up and all muscles in them were sore for several days. I did use a helper carry strap, but it did not help much.

So in a way, yes, this computer is portable. It's not really designed for it, the actual transport process is a long painful slog and if you make an accidental misstep and bump it against a wall you run the risk of breaking everything inside. But it is a "portable computer" as a single person can carry it from one place to another using nothing but their own muscles.

Tying this to software portability

There is a lot of software out there that claims to be "portable" but can only be said to be that in the same way as the computer shown above is "portable". For the rest of the post we're only going to focus on portability to Windows.

Let's say a project has a parser that is built with Lex and Bison. Thus you need to have those programs during compilation. Building them from source is problematic on Windows (because of, among other things, Autotools) so it would be nice to get some prebuilt binaries for them. After a bit of googling you might find this page which provides Windows binaries. That has last been updated in 2004. So no.

You could also install Msys2 and get the binaries with Pacman. If you are using Visual Studio and just want to build the thing, installing a whole separate userspace system and package manager just to get two executables seems like bit of an overkill. Thinking about it further you might realize that you could install Msys on some other machine, get the executables, copy them and their direct dependency DLLs to your machine and put them in PATH. If you try this, the binaries segfault on run, probably because they can't access their localisation files that are "somewhere". 

Is this piece of software portable to Windows? Yes it is, in the "desktop PC is portable" sense, but definitely not in the "a laptop is portable" sense.

As an another example let's look at the ICU project. It claims to be highly portable, and it kind of is, here is a random snippet from their highly portable Makefile.

I don't know about you, but just looking at that … thing gives me a headache. If you ever need to do something the existing functionality does not provide, then trying to decipher what that thing is doing is an exercise in masochism. For Windows this is relevant, because Visual Studio only ships with nmake, which is not at all compatible with Make so you need to decrypt absolutely everything.

Again, this is portable to Windows, you just need to prebuild it with MinGW or using the provided VS solution files, copying the libraries from one place to another and using them. This is very much "desktop PC portable" again.

Sometimes you don't get even that. Take for example the liblangtag project. It is a fairly typical dependency library that provides a single shared library. It even hides its symbols and only exports those belonging to the public API. Sadly it does this using Libtool magic postprocessing. On Windows you have to annotate exported symbols with magic markers. Thus it is actually impossible to build a shared library properly on VS without making source code changes[1]. Thus you have to go the Mingw build route here. But that is again "portable" as in if you spend a ton of time and effort then you can sorta kinda make it work in a rubegoldbergesque way.

Being more specific

Due to various reason I have had to deal with the innards of libraries of different vintage. Fairly often the experience has been similar to dragging my desktop computer across town: arduous, miserable and exhausting. It is something I would wish upon my worst enemy and also upon most of my lesser enemies. It would serve them right.

In my personal opinion saying that some piece of code is portable should imply some basic ease of use. If you need to spend time fighting with it to make it work on an uncommon platform, toolchain or configuration then the thing is not really portable. It also blocks adoption, because if some library is a massive pain to use, people will prefer to reimplement the functionality or use some other library, just to get away from the pain.

Since changing the generally accepted meanings of words is unlikely to work, this won't happen. So in the mean time when you are talking about portability with someone else, do be sure to specify whether you mean "portable as in a desktop gaming PC" or "portable as in a laptop".

How this blog post came about

Some years ago I ported a sizable fraction of LibreOffice to build with Meson. It worked only on Linux as it used system dependencies. I rebased it to current trunk and tried to see if it could be built using nothing but Visual Studio by getting dependencies via the WrapDB. This repo contains the code, which now actually does build some code including dependencies like libxml, zlib and icu.

The code that is there is portable in the laptop sense. You only need to do a git checkout and start the build in a VS x64 dev tools prompt. It does cheat in some points, such as using pregenerated flex + bison sources, but it's not meant to be production quality, just an experiment.

[1] The project in question seems to have more preprocessor macro magic definitions than actual code, so it is possible there is some combination of defines that makes this work. If so, I did not manage to find it. This is typical in many old school C projects.

This year receive the gift of a free Meson manual

About two years ago, the Meson manual was published and made available for purchase. The sales were not particularly stellar and the bureaucracy needed to keep the sales channel going took a noticeable amount of time and effort. The same goes for keeping the book continually up to date.

Thus it came to pass that sales were shut down a year ago. At the time there were some questions on whether the book could be made freely available. This was not done, as it would not really have been fair to all the people who paid actual money to get it. So the book has been unavailable since.

However since an entire year has passed since then, the time has come. I'm making the full PDF manual available for personal use. You can download your own copy via this link. The contents have not been updated in more than a year, so it's not really up to date on details but the fundamentals are still valid.

Enjoy, and have a happy whatever-it-is-that-you-call-the-holiday-at-this-time-of-year.

The boring small print

Even though the book is freely downloadable it is not under any sort of an open license. You can download it and read it for personal use, but redistribution of any kind is not permitted.

Typesetting a whole book part III, the analog edition

In earlier editions (part 1, part 2) we looked at typesetting a full book to a PDF file. This is fun and all, but until you actually hold a physical copy in your hands you don't really know how good the end result is. Puddings, eatings and all that.

So I decided to examine how would you go about printing and binding an entire book. For text I used P. G. Wodehouse's The Inimitable Jeeves. It has roughly 220 pages which is a good amount for perfect binding. Typesetting it in LibreOffice only took a few hours. To make things even simpler I used only one font, the Palatino lookalike P052 that comes packaged with Ghostscript. As the Jeeves stories take place in the 1920s something like Century would have been more period accurate but we'll have to work with what we got.

The only printer I had access to was an A4 laser printer that could only print on one side of the page. Thus to keep things as simple as possible the page size became A5, which is easy to obtain by folding A4 paper in half. None of the printer dialogs seemed to do the imposition I needed (single page saddle fold, basically) so I had to convert the A5 originals to A4 printable sheets with a custom Python script (using PyPDF2)

Printing turned out to have its complications as it always does. The printer did not have Cups drivers so I had to print using Windows. I used Acrobat Reader as it should be the gold standard of PDF manipulation. The original documents were in A4 and the printer only had A4 paper capability. And yet, Acrobat insisted on compositing the pages to 8.5x11 and then printing them distorted because AMERICA! After going through every menu (especially the hidden ones) I managed to make it stop doing that. Then I printed the pages by first printing the odd pages, taking the output, adding it back in the paper tray and then printing the even pages. If you want to try the same, note that for the latter batch you need to print the pages in reverse order.

Do this in small batches. I did and was happy that I did, because every now and then I botched it up and had to redo the pages. But eventually I did prevail.

Then you fold each page in half and stack them to get the text block.

Fasten them together using clips and cut out a piece of cardboard for the cover.

Then glue the spine together, first the pages and then bookbinder's mull on top of that. This step is a bit involved, but Youtube has several videos that show you how to do it.

Then glue on the cover and place the whole thing into a press. Low tech solutions work surprisingly well. (Trivia: this box contains 25 Meson mugs.)

Leave like this overnight. In the morning the book is ready for trimming.

Sadly I don't have a heavy duty paper guillotine so the edges remained a bit rough. Other than that the end result looks pretty nice.

All in all this was a surprisingly simple and fun project. The binding process was so simple that even a craft-noob like myself could get it done without too much trouble. I did make a practice run with some empty pages, though. The clips were a bit fiddly, having a proper press or a woodworking bench would have made the end result smoother.

A call for more downstream testing of Meson

As Meson gets more and more popular, the number of regressions also grows. This is an unvoidable fact of life. To minimize this effort we publish release candidates before the actual releases. Unfortunately not many people use these so many issues are not found until after the release (as happened with 0.60.0).

For this reason we'd like to ask more people to test these rcs on their systems. It's fairly straightforward.

Testing individual projects

If you have a CI that installs Meson using pip, this is easy. You can tell Pip to use prerelease versions with the --pre flag.

pip install --pre meson

If you use prebuilt images rather than reinstalling on every build, do update your images once a week. Meson releases happen traditionally on Sunday evenings European time.

Testing if you are a distro or similar

The release candidates are packaged and uploaded to Debian experimental, so if you can use those, it is the simplest solution. They are not uploaded to unstable as I was instructed not to do so because of breakage potential. If you are a Debian person and know that the above explanation is incorrect and that I should be doing something else, let me know so I can change how that is done.

If you have some different setup that has a full CI run (hopefully something smaller than a full Debian archive rebuild) then doing that with the rc version would be the best test.

If you don't have such a test suite, you'll probably want to set one up for other reasons as well. :)

Glyphtracer 2.0

Ages ago I wrote a simple GUI app called Glyphtracer to simplify the task of creating fonts from scanned images. It seems people are still using it. The app is written in Python 2 and Qt 4, so getting it running becomes harder and harder as time goes by.

Thus I spent a few hours porting it to Python3 and PyQt 5 and bumped the major version to 2.0. The program can be obtained from this Github repo.

Swappable faceplates for laptops

On the whole, laptops are boring. There are three different types.

1. Apple aluminium laptops that all look the same.
2. Other manufacturer's laptops that try to look like Apple laptops that look all the same.
3. RGB led studded gamer laptops that all look the same.

There is a cottage industry that manufactures stickers that are the size of the entire laptop but those are inconvenient and not particularly slick. You only have one chance to apply and any misalignments or air bubbles are going to be there forever.

But what if some laptop manufacturer chose to do things differently and designed their laptops so that the entire face plate were detachable and replaceable? This would allow everybody to customize their own laptops in exactly the way they want to (this should naturally extend to the inner keyboard plate, but we'll ignore that for now). For example you could create a laptop with the exact Pantone color that you want rather than what the laptop manufacturer saw fit to give you.

Or maybe you'd like to have a laptop cover which lights up not the manufacturer's branding but instead the logo of your favourite band?

Or perhaps you are a fan of classic muscle cars?

The thing that separates these covers from what we currently have is that they would not be pictures of things or stickers. The racing stripe cover could be manufactured in exactly the same way as real car hoods. It would shine like real metal. It would feel different when touched. You could, one imagines, polish it with real car wax. It would be the real thing. Being able to create experimental things in isolation makes all sorts of experimentation possible so you could try holograms, laser engravings, embossings, unusual metals and all sorts of funky stuff easily. In case of failure the outcome is just a lost cover plate you can toss into the recycling bin rather than a very expensive laptop that is either unusably ugly or flat out broken.

But wait, there's more. Corporate PR departments should be delighted with this. Currently whenever people do presentations in conferences their laptops are clearly visible and advertise the manufacturer. The same goes for sales people visiting customers (well, eventually, once Covid 19 passes) and so on. Suppose you could have this instead:

Suddenly you have reclaimed a ton of prime advertising real estate that can be used for brand promotion (or something like that, not really a PR person so I'm not intimately familiar with the terminology). If you are a sufficiently large customer and order thousands of laptops at a time, it might be worth it to get all the custom plates fitted at the main factory during assembly. Providing this service would also increase the manufacturer's profit margin.

Hercule Poirot and the Mystery Box, as written by JJ Abrams

Outside, the scorching heat of the British summer could not have made a starker contrast with the icy cold atmosphere of the main living room of Lord Ellington's mansion. All fifteen people sitting around it were staring at the plump Belgian private inspector, who seemed to take great pleasure in waxing his mustache and letting the audience simmer in their own sweat. Finally he spoke.

"You may have wondered why I have called all you here. It has to do with the untimely death of Lady Sybill. We have reason to believe that the three dozen stab wounds in her back were not an accident but that she was in fact ..."

He paused for effect.

"... murdered."

Madame Smith shrieked in terror and dropped her tea cup, which shattered into a million trillion pieces, each of which glittered with the sparkle of a thousand suns. Which was strange, since the curtains were closed and there were no sources of direct light in the room.

"Exactly, mon frére" he said to her, even though his mother tongue was French and he should have known not to call a woman "my brother". And also to spell his sentences properly. But who cares about trivial details such as these when there is a mystery afoot?

"And furthermore, I'm happy to tell you all that our investigation on this has finally come to a conclusion."

Poirot took off his monocle, polished it and then put it back on.

"Oui, we can finally reveal that the identity of the murderer ... will remain unknown forever. Even with our plethora of clues, we could not determine what actually happened. Whoever the murderer was, they got away scott free as we are shutting down the investigation. Mon dieu, that makes this our seventy-sixth unsuccessful murder investigation in a row!"

The audience in the salon seemed perplexed, so captain Hastings chose to interject.

"It could also be space aliens!"

"My money is on an interdimensional time travel accident" countered Inspector Japp with the calm voice of an experienced police officer.

For a while, the room was enveloped in silence. Then the house was blown up by nazis.

Looking at building O3DE with Meson, part II

After the first post, some more time was spent on building O3DE with Meson. This is the second and most likely last post on the subject. Currently the repository builds all of AzCore basic code and a notable chunk of its Qt code. Tests are not built and there are some caveats on the existing code, which will be discussed below. The rest of the conversion would most likely be just more of the same and would probably not provide all that much new things to tackle.

Code parts and dependencies

Like most projects, the code is split into several independent modules like core, testing, various frameworks and so on. The way Meson is designed is that you traverse the source tree one directory at a time. You enter it, do something, possibly recurse into subdirectories and then exit it. Once exited you can never again return to the directory. This imposes some extra limitations on project structure, such as making circular dependencies impossible, but also makes it more readable.

This is almost always what you want. However there is one exception that many projects have: the lowest layer library has no internal dependencies, the unit testing library uses that library and the tests for the core library use the unit testing library. This is not a circular dependency as such, but if the unit tests are defined in the same subdir as the core library, this causes problems as you can't return to it. This needs to be broken in some way, like the following:

subdir('AzCore')
subdir('AzTest')
subdir('AzCore/tests')

Code generation

Most large projects have a code generator. O3DE is no exception. Its code generator is called AutoGen and it's a Python script that expands XML using Jinja templates. What is strange is that it is only used in three places, only one of which is in the core code. Further, if you look at the actual XML source file it only has a few definitions. This seems like a heavy weighted way to go about it. Maybe someone could summon Jason Turner to constexrpify it to get rid of this codegen.

This part is not converted, I just commented out the bits that were using it.

Extra dependencies

There are several other dependencies used that seem superfluous. As an example the code uses a standalone library for MD5, but it also uses OpenSSL, which provides an MD5 implementation. As for XML parsers, there are three, RapidXML, Expat and the one from Qt (though the latter is only used in the editor).

Editor GUI

Almost all major game engines seem to write their own GUI toolkits from scratch. Therefore it was a bit surprising to find out that O3DE has gone all-in on Qt. This makes it easy to use Meson's builtin Qt 5 support, though it is not without some teething issues. First of all the code has been set up so that each .cpp file #includes the moc file generated from its header:

#include "Components/moc_DockBarButton.cpp"

Meson does things differently and builds the moc files automatically so users don't have to do things like this. They are also written in a different directory than what the existing configuration uses so this include could not work, the path is incorrect. This #include could be removed altogether, but since you probably need to support both at the same time (due to, for example, a transition period) then you'd need to do something like this:

#ifndef MESON_BUILD

#include "Components/moc_DockBarButton.cpp"

#endif

What is more unfortunate is that the code uses Qt internal headers. For some reason or another I could not make them work properly as there were missing private symbols when linking. I suspect that this is because distro Qt libraries have hidden those symbols so they are not exported. As above I just commented these out.

The bigger problem is that O3DE seems to have a custom patches in their version. At least it refers to style enum values that do not exist. Googling for the exact string produces zero relevant matches. If this is the case then the editor can not be used with official Qt releases. Further, if said patches exist, then they would need to be provided to the public as per the LGPL, since the project is providing prebuilt dependency binaries. As mentioned in the first blog post, the project does not provide original sources for their patched dependencies or, if they do, finding them is not particularly easy.

What next?

Probably nothing. It is unlikely that upstream would switch from CMake to Meson so converting more of the code would not be particularly beneficial. The point of this experiment was to see if Meson could compile O3DE. The answer for that is yes, there have not been any major obstacles. The second was to see if the external dependencies could be provided via Meson's Wrap mechanism. This is also true, with the possible exception of Qt.

The next interesting step would be to build the code on multiple platforms. The biggest hurdle here is the dependency on OpenSSL. Compiling it yourself is a bear, and there is not a Wrap for it yet. However once this merge request is merged, then you should be able to build OpenSSL as a Meson subproject transparently. Then you could build the core fully from source on any platform.

How much effort would it take to convert OpenSSL's Perl source code generators to Python?

There is an ongoing discussion to write Meson build definitions to OpenSSL so it can be added to the WrapDB and built transparently as a subproject. One major issue is that OpenSSL generates a lot of assembly during build time with Perl. Having a Perl dependency would be bad, but shipping pregenerated source files would also be bad. Having "some pregenerated asm" that comes from "somewhere" would understandably be bad in a crypto library.

The obvious third option would be to convert the generator script from Perl to Python. This is not the first time this has been proposed and the counterargument has always been that said conversion would take an unreasonable amount of effort and could never be done. Since nobody has tried to do the conversion we don't really know whether that claim is accurate or not. I converted the x86_64 AES asm generator to see how much work it would actually take. The code is here.

The code setup

The asm generator has two different levels of generators. First a Perl script generates asm in a specific format and after that a second Perl script converts it to a different asm type if needed (Nasm, At&T, Intel etc). The first script can be found here and the second one is here. In this test only the first script was converted.

At first sight the task seems formidable. The generator script is 2927 lines of code. Interestingly the asm file it outputs is only 2679 lines. Thus the script is not only larger than its output, it is also less understandable than the thing it generates both because it is mishmash of text processing operations and because it is written in Perl.

Once you get over the original hump, things do get easier. Basically you have a lot of strings with statements that look like this:

movzb `&lo("$s2")`,$acc0

This means that the string needs to be evaluated so that $s2 and $acc0 are replaced with the values of variables s2 and acc0. The backticks mean that you have to then call the lo function with the given value and substitute that in the output string. This is very perlistic and until recently would not have been straightforward to do in Python. Fortunately now we have f-strings so that becomes simply:

f'movzb {lo(s2)},{acc0}'

With that worked out the actual problem is no longer hard, only laborious. First you replace all the Perl multiline strings with Python equivalents, then change the function declarations from something like this:

sub enctransform_ref()
{ my $sn = shift;

to this:

def enctransform_ref(sn):

and then it's a matter of repeated search-and-replaces for all the replacement variables.

The absolute best thing about this is that it is almost trivial to verify that the conversion has been done without errors. First you run the original script and store the generated assembly. Then you run your own script and compare the output. If they are not identical then you know exactly where the bug is. It's like having a unit tests for every print statement for the program.

How long did it take?

From scratch the conversion took less than a day. Once you know how it's done a similar conversion would take maybe a few hours. The asm type converter script seems more complicated so would probably take longer.

A reasonable port would contain these conversions for the most popular algorithms to the most popular CPU architectures (x86, x86_64, arm, aarch64). It would require a notable amount of work but it should be measured in days rather than months or years. I did browse through some of the other asm files and it seems that they have generators that work in quite different ways. Converting them might take more or less work, but probably it would still be within an order of magnitude.

A quick look at the O3DE game engine and building it with Meson

Earlier today I livestreamed what it would take to build a small part of the recently open sourced O3DE game engine. The attempt did not get very far, so here is a followup. It should not be considered exhaustive in any way, it is literally just me poking the code for a few hours and writing down what was discovered.

Size

The build system consists of 56 thousand lines of CMake.

Name prefixes

The code has a lot of different naming convention and preserved prefixes. These include "Cry" which seem to be the oldest and come from the original CryEngine. The most common one is "az", which probably stands for Amazon. The "ly" prefix probably stands for "Lumberyard" which was the engine's prior name. Finally there is "PAL", which is not really explained but seems to stand for "platform abstraction layer" or something similar.

Compiling

The code can be obtained from the upstream repo. There's not much more you can do with it since it does not actually build on Linux (tested on latest Fedora) but instead errors out with a non-helpful CMake error message.

When I finally managed to compile something, the console practically drowned in compiler warnings. These ranged from the common (missing virtual destructors in interface classes) to the bizarre (superfluous pragma pops that come from "somewhere" due to macros).

Compiler support

The code explicitly only supports Visual Studio and Clang. It errors out when trying to build with GCC. Looking through the code it seems like it is mostly a case of adding some defines. I tried that but pretty quickly ran into page-long error messages. A person with more knowledge of the inner workings of GCC could probably make it work with moderate effort.

Stdlib reimplementations

O3DE is based on CryEngine, which predates C++ 11. One place where this shows up is that rather than using threading functionality in the standard library they have their own thread, mutex, etc classes that are implemented with either pthread or Windows threads. There may be some specific use cases (thread affinity?) why you'd need to scoop to plain primitives but otherwise this seem like legacy stuff nobody has gotten around to cleaning.

Yes, there is a string class. Several, in fact. But you already knew that.

Dependencies

This is where things get weird. The code uses, for example, RapidXML and RapidJSON. For some reason I could not get them to work even though I used the exact same versions that are listed in the CMake definition. After a fair bit of head scratching things eventually became clear. For example the system has its own header for RapidXML called rapidxml.h whose contents are roughly the following:

#define RAPIDXML_SKIP_AZCORE_ERROR

// the intention is that you only include the customized version

// of rapidXML through this header, so that

// you can override behavior here.

#include <rapidxml/rapidxml.h>

Upstream does not provide its header in a rapidxml subdirectory, it is under include. The same happens when the header is installed on the system. Thus the include as given can not work. Even more importantly, the upstream header is not named rapidxml.h but instead rapidxml.hpp.

It turns out that O3DE has its own dependency system which takes upstream source, makes arbitrary changes to it, builds it and then is provided as a "package" which is basically a zip file with headers and prebuilt static libs. These are downloaded from not-at-all-suspicous-looking URLs like this one. What changes are done to these packages is not readily apparent. There are two different repos with info but they don't seem to have everything.

When using external libraries like this there are two typically two choices. Either you patch the original to "fit in" with the rest of your code or you can write a very small adapter wrapper. This project does both and with preprocessor macros no less.

The whole dependency system seems to basically be a reimplementation of Conan using pure CMake. If that sentence on its own does not make cold sweat run down your back then let it be noted that one of the dependencies obtained in this way is OpenSSL.

The way the system is set up prevents you from building the project using system dependencies. This includes Qt as the editor GUI is written with it. Neither can you build the entire project from source yourself because the existing source only works with its own special prebuilt libraries and the changes applied do not seem to be readily available as patches.

Most of this is probably because CryEngine was originally written for internal use on Windows only. This sort of an approach works for that use case but not all that well for a multiplatform open source project.

Get the code

My experimental port that compiles only one static library (AzCore) can be downloaded from this Github repo. It still only supports Clang but adding GCC support should be straightforward as you don't need to battle the build mechanism at least.

A followup to the Refterm blog post

My previous blog post was about some measurements I made to Refterm. It got talked about on certain places on the net from where it got to Twitter. Then Twitter did the thing it always does, which is to make everything terrible. For example I got dozens of comments saying that I was incompetent, an idiot, a troll and even a Microsoft employee. All comments on this blog are manually screened but this was the only time when I just had to block them all. Going through those replies  seemed to indicate that absolutely everyone involved [1] had bad communication and also misunderstood most other people involved. Thus I felt compelled to write a followup explaining what the blog post was and was not about. Hopefully this will bring the discussion down to a more civilized footing.

What we agree on

Let's start by writing down all the things we agree on.

1. The current Windows terminal is slow.
2. It can be made faster.
3. A GPU-based renderer (such as the one in Refterm) can render terminal text hundreds of times faster than the current implementation in Windows terminal.

Note that even the people working on Microsoft Terminal acknowledged all of these to be true before any code on Refterm had been written. From what I can tell, #3 is what Refterm set out to prove and it was successfull at it.

So what's the issue then?

Once the code was out many people started making these kinds of statements.

Windows terminal should switch to using this method because it is obviously superior and not doing that is just laziness and excuses.

Now, and this is important, the original people who worked on Refterm never made these kinds of claims. They did not! And further, I never claimed that they did. Other people ("the talking heads on the Internet") made those claims and then mental misattribution took over. This is unfortunate but sadly almost inevitable whenever these kinds of debates happen. That then leads to the obvious follow up question:

Could the rendering mechanism used in Refterm be put in Windows terminal proper? If not, why not? If yes, what would the outcome be like and would the code need changing?

This is what my original blog post was about. Since this was outside of the original project's design goals I should have stated the goals out explicitly. I did not, and that is a flaw on my part, sorry about that.

The problems with prototypes

Implementing a simple prototype of an existing program (or a part of it), achieving great success and then extrapolating from that to the whole program (and to reiterate: the original Refterm authors did not do this speculation, other people did) has a well known pitfall. I could write an entire blog post about it. Fortunately I don't have to since Matthew Garrett already wrote one. I recommend that everyone reads that before continuing with this post.

The tl/dr version is that when you bring a protype up to sufficient feature parity with an existing implementation you will encounter unexpected problems. The algorithms or approaches you took might not be feasible. You might need to add a bunch of functionality that you never considered or had even heard of. Until you have the entire implementation you don't know whether you approach will work. In fact you can't know it. Anyone who claims to know is lying, either to others or to themselves. (Reminder again: the Refterm authors did not make these kinds of estimates.)

We can try to come up with some of the obstacles and problems one could have when moving the prototype implementation into a real one and then examine those. They can't prove fitness but they can reveal un-fitness. The points discussed in the blog post were just some that I came up with. There are undoubtedly many others.

Resource usage

Let's start this by acknowledging a notable flaw in the original post [2]. When evaluating memory usage the post only compared it against other types of apps, not other terminals. I ran some new measurements by starting both the Windows cmd.exe shell as well as the Git-Scm's MSYS2 terminal, running a few simple commands and looking at memory consumption with the Task Manager. Refterm took 350 MB of ram, MSYS2 took 4 MB and cmd.exe took 7 MB.

People really love their terminals. I have seen setups where a single developer has 10+ different terminals open at the same time all as different processes (with several tabs each). So even if 300 MB of ram usage for a single app would be fine, using 3 GB of ram in this case would not be. Thus one would either need to dramatically reduce memory usage or have something like a shared glyph cache between all the various processes. That requires shared GPU resources, some sort of an IPC communication mechanism, multiprocess cache invalidation and all other fun stuff (or that is my understanding at least as a Windows and GPU neophyte, if there is a simpler way do let me know).

This piece of information is useful and important on its own. It gives new information and understanding of what the code does and does not do.

A retort to this that was actually made by Refterm developers was that "there are variables and knobs in the code you can tweak to affect the performance". To this I say: no. The first tests should always be done with the exact setup the code ships with. There are two reasons for this. First of all, that makes experiments made by different people directly comparable with each other. Secondly, the original author(s) know the code best and thus it makes sense to choose those parameter values that they went with. 

Code layout and nonstandardness

Let's start again with the thing we all agree on:

For your own projects you can choose whatever code layout, build system, organization and so on that you want. Do whatever works best for you and don't let anyone tell you otherwise!

Things get more complicated when you start including other people, especially outside your own circle of devs. An open source project is a typical example. An anonymous commenter told me the following:

This is also the simplest possible code structure, very simple to work with for new contributors.

This sentence is interesting in that it is both true and not true. If you have a person who has no prior programming knowledge then yes, the layout is simplest possible and easy to get started with. On the other hand if the potential contributor is already accustomed to the "standard way of setting up a project" then things change and the nonstandard layout can get confusing [3] and can be a barrier to entry for new people. This is the nature of teamwork: sometimes it might make sense to do the thing that is inconvenient for you personally for the benefit of others. Sometimes it does not make sense. Like most things in life being nonstandard is not an absolute thing, it has its advantages but also disadvantages.

I actually encountered a technical disadvantage caused by this. I wanted to compile and run Refterm under Address Sanitizer, which is a really great tool for finding bugs. Asan is integrated into the latest VS and all you need to do is to add /fsanitize=address flags to the compiler to use it. This does not work for Refterm but instead leads to a bunch of linker errors. The Asan library depends on the C runtime and Refterm is explicitly set up not to use it. It took me a fair bit of time to work out that the way to get it working is to go through the code and replace the WinMainCRTStartup function with a "normal" WinMain function and then the linker would do the right thing [4].

That SIMD memcpy thing

I pondered for a while whether I should mention the memcpy thing and now I really wish I hadn't. But not for the reasons you might think.

The big blunder I did was to mention SIMD by name, because the issue was not really about SIMD. The compiler does convert the loop to SIMD automatically. I don't have a good reference, but I have been told that Google devs have measured that 1% of all CPU usage over their entire fleet of computers is spent on memcpy. They have spent massive amounts of time and resources on improving its performance. At least as late as 2013, performance optimizing memcpy was still subject to fundamental research (or software patents at least). For reference here is the the code for the glibc version of memcpy, which seems to be doing some special tricks.

If this is the case and the VS stdlib provides a really fast memcpy then rolling your own does cause a performance hit (though in this particular case the difference is probably minimal, possibly even lost in the noise). On the other hand it might be that VS can already optimize the simple version to the optimal code in which case the outcome is the same for both approaches. I don't know what actually happens and finding out for sure would require its own set of tests and benchmarks.

Concluding and a word about blog comments

That was a very long post and it did not even go through all the things I had in mind. If you have any comments, feel free to post them below, but note the following:

• All comments are prescreened and only appear after manually approved.
• Any comments that contain insults, whining, offensive tone or any other such thing will be trashed regardless of its other merits
• The same goes for any other post whose contents makes it obvious that the commenter has not read the whole text but is just lashing out.

[1] Yes, this includes me. It most likely includes you as well.

[2] Thanks to an anonymous commenter for pointing this out.

[3] I can't speak for others, bu it was for me.

[4] There may have been other steps, but this was the crucial one.

Looking at the performance of Refterm

Recently a known strong-opinion-holder Casey Muratori wrote a reference implementation for a new fast terminal renderer.  The goal is (as far as I can tell) to implement the same functionality as existing terminals using a lot less resources. The code is here and according to the readme it supports, for example:

• All of Unicode, including combining characters and right-to-left text like Arabic
• Line wrapping
• Reflowing line wrapping on terminal resize

The tests they perform show that the new terminal works several orders of magnitude faster than the default terminals in Windows. Seems nice, so let's do a simple code review to see how it actually stacks up.

Code setup

The code is arranged in a very non-standard way. There are two main source files, one C and one C++ that are built with a BAT file. Those files first #define a bunch of stuff and then #include all other source code files (not header files, source files) directly. This is very old school and against pretty much any recommended best practice for C. This works will for single-person projects but is a big hurdle for any new contributors.

The build uses the /GS- /Gs999999set command line arguments, which disable security features. This seems ill-advised for a terminal application, whose main job is to parse untrusted input. All libraries used are not defined in the build file but instead as pragmas inside the source files. The program also does not link the C library and because of this has its own simple implementations of memcpy and memset. This means you don't get the SIMD-optimized versions from the stdlib (the performance impact of this was not tested).

Resource usage

Resource consumption was measured by checking out the code, building it with the bat as instructed by upstream, starting the program and letting it idle. This is how it appears in Windows' task manager.

The app uses 0.5% of CPU and a whopping 14% of GPU just to display a blinking cursor. This could be said to be not particularly resource efficient. This is probably due to the fact that there is no rate limiter (or VSYNC) so the app just spams the system all the time. The true resource usage can't be meaningfully compared until this is fixed.

What can be measured, though, is memory usage. As can be seen in the image [1] the Refterm application uses 351 MB of memory when idle (the test war run using a 4k monitor). Based on discussions on the Internet, an acceptable amount of memory usage for a terminal is around 10-20 MB. Refterm uses 10x as much. In fact, as you can tell, running two instances of Refterm takes more memory than a fully blown Firefox with tens of open tabs. For comparison I also tested the Spotify app which is implemented in Electron. When playing music it only took ~150 MB, less than half of an idling Refterm.

Reliability

A terminal widget is a high risk app because it has to parse a lot of untrusted text. Thus it is very important that it tolerates malformed and malicious input. A simple test is to start Refterm and then run splat refterm_debug_msvc.pdb in Refterm. This basically dumps 1.3 MB of binary data to the terminal. This causes Refterm to immediately freeze and take 100% CPU. The app window can not be closed and can only be killed via the task manager.

Conclusions

In its current form Refterm can, at most, be seen as a proof of concept which can not be reasonably compared against a full-featured terminal renderer. It is neither memory-efficient nor reliable against malformed input.

[1] I don't know much about Windows development, so I don't know how representative this number is of the "real" resource usage. At least on Linux the output of top should be taken with a grain of salt. I also tried VS's profiler and it claimed that the app took over 450 MB of ram.

Update 2021/7/7

This blog post got linked to from places which caused a flood of mostly nonproductive and toxic comments. Because of this no further comments are allowed for this post. Sorry.

Typesetting a full book part II, Scribus

Some time ago I wrote a blog post on what it's like to typeset an entire book using nothing but LibreOffice. One of the comments mentioned that LO does not do a great job of aligning text. This is again probably because it needs to copy MS Word's behaviour, which means greedy line splitting. Supposedly Scribus does this a lot better, but the only way to be really sure was to typeset the whole text with Scribus. So that's what I did (using the latest 1.5 release from Flathub).

Workflow for Scribus

Every program has the things it is good for and things it's not that good for. Scribus' strengths lie in producing output with fairly short pieces of text with precise layout requirements, especially if there are many images. A traditional "single flow of text" is not that, so there are some things you need to plan for.

First of all, a Scribus document should not be created until the text is almost completely finished. Doing big changes (like adding text to existing chapters, changing physical page size etc) can become quite tedious. Scribus also does not do long pieces of text particularly smootly. I tried loading all 350is pages to a single linked frame sequence. It sort of worked, but things got quite laggy quite quickly. Eventually I converged on a layout where every chapter was its own set of linked frames. The text was imported directly from LO files that held one chapter each. The original had just one big LO file, so I had to split it up by hand for the import. If the original had been done with master documents, this would have been simpler.

The table of contents had to be done by hand again. Scribus has support for tables, but they could not be used, because tables drew outlines around each cell and I could not find a way to switch that off. Websearching found several pages with info, but none of them worked. It also turns out that you can not add page references to table cells, only to text frames. No, I don't know why either. The option was greyed out in the menus and trying to sneakily copypaste a page reference from a text frame to a table caused a segfault.

Issues discovered

While LO was surprisingly bug free, Scribus was less so and I encountered many bugs and baffling missing features, such as:

• Scribus would sometimes create empty text frames far outside the document (i.e. to page 600 on a 300 page document)
• Text frames got a strange empty character at their end which would cause text overflow warnings, deleting it did not help as the empty characters kept reappearing
• Adding a page reference to an anchor point would always link to the page where the linked frame sequence started, not where the anchor was placed
• Text is not hyphenated automatically, only by selecting a text frame and then selecting extras > hyphenate text in the main menu, one would imagine hyphenation being a paragraph style property instead
• I managed to create an anchor point that does not exist anywhere except the mark list, but deleting it leads to an immediate segfault

None of these obstacles were insurmountable, but they made for a nonsmooth experience. Eventually the work was done and here is how they compare (LO on the left, Scribus on the right).

As you can probably tell, Scribus creates more condensed output. The settings were the same for both programs (automatically translated from LO styles by Scribus, not verified by hand) and LO's output file was 339 pages compared to 326 for Scribus.

Which one should you use then?

Like most things in life, that depends. If your document has a notable amount of mathematics, then you most likely want to go with LaTeX. If the document is something like a magazine or you require the highest typographical quality possible, then Scribus is a good choice. For "plain old books" the question becomes more complicated.

If you need a fully color managed workflow, then Scribus is the only viable option. If the default output of LO is good enough for you, the document has few figures and you are fine with needing to have a great battle at the end to line the images up, LO provides a fairly smooth experience.  You have to use styles properly, though, or the whole thing will end up in tears. LO is especially suitable for documents with lots of levels, headings and cross references between the two. LaTeX is also very good with those, but its unfortunate downside is that defining new styles is really hard. So is changing fonts, so you'd better be happy with Computer Modern. If the document has lots of images, then LaTeX's automatic figure floats make a ton of manual work completely disappear.

Original data

The original source documents as well as the PDF output for both programs can be found in this Github repo.