Please provide tarball releases of your projects

A recent trend in open source projects seems to be to avoid releasing proper release archives (whether signed with GPG or not). Instead people add Git tags for release commits and call it a day.

A long and arduous debate could be had whether this is "wrong", "right" and whether Git hashes are equivalent to proper tarballs or not, or if --depth=1 is a good thing or not. We're not going to get into that at all.

Instead I'd like to kindly ask that all projects that do releases of any kind to provide actual release tarballs for the following two reasons:

1. It takes very little effort on your part.
2. Proper release archives make things easier for many people consuming your project.

This makes sense just from a pure numbers game perspective: a little work on your part saves a lot of work for many other people. So please do it.

What about Github automatic archive generation?

Github does provide links to download any project commit (and thus release) as an archive. This is basically a good thing but it has two major issues.

1. The filenames are of type v1.0.0.tar.gz. So from a file name you can't tell what it contains and further if you have two dependencies with the same version number, the archive files will have the same name and thus clash. Murphy's law says that this is inevitable.
2. The archives that Github generates are not stable. Thus if you redownload them the files may have different checksums. This is bad because doing supply chain verification by comparing hashes will give you random failures.

The latter might get fixed if Github changes their policy to guaranteed reproducible downloads. The former problem still exists.

A simple webui-only way to do it

If you don't want to use git archive to generate your releases for whatever reason, there is a straightforward way of doing the release using only the web ui.

1. Create your release by tagging as usual.
2. Download the Github autogenerated tarball with a browser (it does not matter whether you choose zip or tar as the format, either one is fine).
3. Rename the v1.0.0.tar.gz file to something like myproject-1.0.0.tar.gz.
4. Go to the project tags page, click on "create a new release from tag".
5. Upload the file from step #3 as a release file.

Supporting external modules in Godot game engine with Meson

The disclaimer

None of this is in upstream Godot yet. It is only a proposal. The actual code can be obtained from the meson2 branch of this repository. Further discussion on the issue should be posted here.

The problem

Godot's code base is split into independent modules that can be enabled and disabled at will. However many games require custom native code and thus need to define their own modules. The simplest way to do this (which, I'm told, game developers quite often do) is to fork the upstream repo and put your your code in it. This works and is a good solution for one-off projects that write all extra code by themselves. This approach also has its downsides. The two major ones are that updating to a newer version of the engine can turn into a rebasing hell and that it is difficult to combine multiple third party modules.

Ideally what you'd want to do is to take upstream Godot and then take a third party module for, say, physics and a second module written by completely different people that does sound effects processing, combine all three and have things just work. Typically those modules are developed in their own repositories. Thus we'd end up with the following kind of a dependency graph.

This would indicate a circular dependency between the two repositories: in order to build the external module you need to depend on the Godot repo and in order to build the Godot you need to depend on the external repository. This is bad. If there's one thing you don't want in your source code it is circular dependencies.

Solving it using Meson's primitives

If you look at the picture in more detail you can tell that there is no circular dependencies between individual targets. Thus to solve the problem you need some way to tell the external dependency how to get the core libraries and headers it needs and conversely a way for the main build to extract from the external project what modules it has built and where they are. As Meson subprojects are built in isolation one can't just blindly poke the innards of other projects as one can do if everything is in a single megaproject.

The way Godot's current build is set up is that first it defines the core libraries, then all the modules and finally things on top of that (like the editor and main executable). We need to extend this so that external modules are set up at the same time as internal modules and then joined into one. Thus the final link won't even be able to tell the difference between external and internal modules.

First we need to set up the dependency info for the core libraries, which is done like this:

godotcore_dep = declare_dependency(include_directories: INCDIRS,
                                   compile_args: CPP_ARGS,
                                   link_args: LINK_ARGS)

meson.override_dependency('godotcore', godotcore_dep)

First we set up a dependency object that encapsulates everything needed to build extension modules and then specify that whenever a dependency called godotcore is looked up, Meson will return the newly defined object. This even works inside subprojects that are otherwise isolated from the master project.

Assuming we have a list of external module subprojects available, we can go through them one by one and build them.

foreach extmod : external_modules

    sp = subproject(extmod)

    MODULE_DEPENDENCIES += sp.get_variable('module_dep')

    MODULES_ENABLED += sp.get_variable('module_names')

endforeach

The first line runs the subproject, the latter two are ignored for now, we'll come back to them. The subproject's meson.build file starts by getting the dependency info.

godotcore_dep = dependency('godotcore')

Then it does whatever it needs to build the extension module. Finally it defines the information that the main Godot application needs to use the module:

module_dep = declare_dependency(link_with: lib_module_tga,

  include_directories: '.')

module_names = ['tga']

In this case I have converted Godot's internal tga module to build as an external module hence the name. This concludes the subproject and execution resumes in the master project and the two remaining lines grab the module build information and name and append them to the list of modules to use.

This is basically it. There are obviously more details needed like integrating with Godot's documentation system for modules but the basic principle for those is the same. With this approach the integration of multiple external modules is simple: you need to place them in the main project's subprojects directory and add their names to the list of external module subprojects. All of this can be done with no code changes to the main Godot repo so updating it to the newest upstream version is just a matter of doing a git pull.

Typesetting an Entire Book Part IV: The Content

In previous blog posts (such as seals this one) we looked into typesetting a book with various FOSS tools. Those have used existing content from Project Gutenberg. However it would be a whole lot nicer to do this with your own content, especially since a pandemic quarantine has traditionally been a fruitful time to write books. Thus for completeness I ventured out to write my own. After a fair bit of time typing, retyping, typesetting, imposing, printing, gluing, sandpapering and the like, here is the 244 page product that eventually emerged from the pipeline.

As you can probably tell, this was the first time i did gouache lettering with a brush. It behaved differently from what I expected and due to reasons I could only do this after the cover had been attached to the text block, so there was no going back or doing it over.

What's its name in English and what genre does it represent?

The first one of these is actually quite a difficult question. I spent a lot of time trying to come up with a working translation but failed. Not only is the title a pun, it contains an intentional spelling error. A literal translation would be something like The First Transmission Giltch, though that misses the fact that the original name contains the phrase First Contact. The working title for the book was Office Space in Space.

As you can probably tell, the book is a sci-fi satire about humanity's first contact with alien civilisations, but it can be seen as an allegory of a software startup company. ISO standardisation also plays a minor part, and so do giant doughnuts, unexpected gravities, a cosplay horse and even space sex (obviously).

At this point many of you have probably asked the obvious question, namely isn't this just a blatant Hitchhiker's Guide to the Galaxy ripoff?

Not really. Or at least I'd like to think that it isn't. This can be explained with an analogy: what Hitchhiker's is to Star Wars, this book is to Star Trek. More specifically it is not high fantasy, but more technical, down to earth and gritty, for lack of a better term.

Can I read it?

You almost certainly can not.

In fact, let's be scientific and estimate how unlikely it would be. The first hurdle is getting the book published. Statistics say that only one book out of a thousand offered to publishers actually gets published. Even if it did get published and you had a physical copy in your hands, you probably still could not read it, since it is written in Finnish, a language that is understood only by 0.1 percent of the planet's population. If we estimate how many people who could read it actually would read it then the chances are again roughly one of a thousand.

Putting all these together and assuming a planetary population of 7 billion we find that the book will only be read by around seven people. Thus far I have convinced five of my friends to read the preview version so there are only two slots available. Your chances of being one of the two are thus quite slim. On the other hand if you are an extreme hipster, the kind who likes to boast to their friends that they only read books in languages they don't speak and which require their readers to understand both French and Latin in order to get some of the jokes within, you may have found your Citizen Kane.

If you can find it, that is.

Converting Godot game engine to Meson, how you can help

There has long been interest in switching the Godot game engine from its current SCons build system to Meson. The actual details are here, but as a quick summary:

• The original port was written by community members.
• It got stale, so I have been keeping it rebased against current master
• The original plan was to review and possibly merge it immediately after the 4.0 alpha release.

Unfortunately the people who should do the review are currently very busy with other things and can't spend much time on this. It would be important to get the change in before the 4.0 release as changing build systems mid-release is apparently a difficult task.

How can you participate?

As the actual devs are busy, what we can do is try to make things as easy for them as possible. In other words the port needs review and testing. The simplest way is to help is to check out the code from the meson branch of this repo, compiling it and running it. Then report your success (or failure) in this Github discussion thread. I will keep rebasing the branch against upstream master at regular intervals.

Note that if you encounter bugs, please do not file them against the upstream project unless you have verified that they also occur with a regular SCons build.

Currently it compiles and runs for me on Linux, Windows and macOS, but the more people can verify it the better.

It would be especially useful if people could test it on Android. FWICT the original uses Android Studio for the Java bits and somehow uses SCons to build a shared library that is used. The Meson build does build the shared lib, but it has not been tested. If someone with Android dev experience could set up and test the whole pipeline it would be great.

The same goes for iOS, though I know even less how it should be set up as I have not really done iOS development. It should build for iOS (there is a cross file in the repo) but FWICT it has never been tested beyond that.

The eventual goal

The aim is to make the Meson port as polished and tested as possible so that once upstream makes their final decision it should be as easy as possible for them (regardless of what they eventually end up choosing). The more people have tried it, the more confidence there is that the port can actually do all the things that are needed.

Compiling LibreOffice with Meson even further

After building the basics of LO on Windows and macOS the obvious next step is to build all of it. This is just grunt work and quite boring at that actually. I almost got it done apart from the fact that at the end I got lazy and skipped those bits that require weird dependencies (of which more later).

The end result has approximately 7800 individual compilation and linking steps. It takes about 30 minutes on a 16 core Ryzen 3700X CPU using Windows and Visual Studio. On a 7 year old 4 core Macbook it takes around 3 hours.

What is still needed to make it work?

Quite a bit, actually. The most pertinent would be all the configuration files that get installed. There are a lot of them and they need to be exactly right, otherwise the end result fails to start with cryptic error messages, if any. Unlike code compilation there is no easy way to know in advance what should be done or how things should behave. Ideally you'd get help with people who know the innards of the program and all the configgen bits.

The next bit is Java. You can compile LO without Java at all and, from what I can tell, there are plans to replace all Java with native code. Unfortunately that will take some time so in the mean time something would need to be done with this. Meson does have native Java support but, as far as I know, it has never been tested with huge projects so there would probably be some teething problems.

There are also many parts of existing native code that has not been ported due to missing dependencies. There many plugins for different SQL server connectors. They all need the respective client libs. These have not been ported.

All of that would handle perhaps 80% of the work. The remaining 80% would be split evenly across a bunch of places from deployment scripts to CI config to workspace changes and so on. There is arguably a third 80% bit, namely convincing people to adopt the new system.

What horrors interesting features were found lurking in the dependencies?

Oh boy, were there some. In the interest of fairness their names have been anonymized for obvious reasons.

One of the dependencies has not had a release in six years. It builds only with Autotools, generates a config.h header and all that. During porting it seemed that sometimes changes to the configuration header are not picked up. After enough digging it turned out that the header is not included anywhere. It is not even force included with a compiler flag. It is 100% useless.

This is problematic because spawning external processes to do all SIZEOF_INT check and the like are by far the slowest part of Meson and not something that can be easily sped up. Even worse, 99% of the time those checks would still be useless today. If you need integers of a specific size, use stdint.h instead. If your library has its own named integer types, define those via entries provided by stdint.h instead of hacking things by hand. It is faster, more reliable and less effort. It even works on Visual Studio since several years ago. There are still reasons to do the manual thing, but they are extremely rare. So unless you are doing something very low level like GLib or have to support Ultrix from 1992 then there really is no reason not to use stdint.h.

Another dependency claims to support Windows but it does not have any symbol visibility exports but instead relies on GCC exporting all symbols by default so it can't be build as a shared library using Visual Studio, only MinGW.

While Autotools is peculiar in many ways, one of the nicest things about it is that the config.h header it creates has a very rigid form. Thus it is a lot easier to create a tool that reads that file and works out backwards what checks it has done. Meson has a tool for this and converting Autoconf checks with it is actually quite simple. Sadly sometimes people feel the need to reinvent their own. One of LO's dependencies does exactly that. Its config header template has the regular defines. It also has lines like this:

${FOOBAR__O_SOMETHINGSOMETHING}

After a lot of inscrutable macro code written in a Certain Make-based system splattered about the source tree, the end result eventually gets expanded to this:

#define internalname func_name_in_current_libc

It is unclear why this was done and what the advantage was, but the downside is clear. There is no automatic way to convert this, it can only be reverse engineered and reimplemented by a human being. This is slow.

In general there are a lot of weird checks in various projects all of which take time to do. For example there is a fairly modernish C++ dependency that checks for the sleep C function rather than using the builtin this_thread::sleep_for function that has existed for over 11 years now. There are also checks for whether the printf function is available (because, you know, some of your users might be living in the year 1974, one can never be sure about these things). There is even a case where this test fails on the Visual Studio compiler.

All of this is to say that I have now gone with this about as far as I'm willing to do on my own. If there are people who want to see this conversion properly happen, now would be the time to get involved.