sunnuntai 12. toukokuuta 2019

Emulating rpath on Windows via binary patching

A nice feature provided by almost every Unix system is rpath. Put simply it is a way to tell the dynamic linker to look up shared libraries in custom directories. Build systems use it to be able to run programs directly from the build directory without needing to fiddle with file copying or environment variables. As is often the case, Windows does things completely differently and does not have a concept of rpath.

In Windows shared libraries are always looked up in directories that are in the current PATH. The only way to make the dynamic linker look up shared libraries in other directories is to add them to the PATH before running the program. There is also a way to create a manifest file that tells the loader to look up libraries in a special place but it is always a specially named subdirectory in the same directory as the executable. You can't specify an arbitrary path in the manifest, so the libraries need to be copied there. This makes Windows development even more inconvenient because you need to either fiddle with paths, copy shared libraries around or statically link everything (which is slooooow).

If you look at Windows executables with a hex editor, you find that they behave much the same way as their unixy counterparts. Each executable contains a list of dependency libraries that it needs, such as helper.dll. Presumably what happens is that at runtime the dynamic linker will parse the exe file and pass the library names to some lookup function that finds the actual libraries given the current PATH value. This raises the obvious question: what would happen if, somehow, the executable file would have an absolute path written in it rather than just the filename?

It turns out that it works and does what you would expect it to. The backend code accepts absolute paths and resolves them to the correct file without PATH lookups. With this we have a working rpath simulacrum. It's not really workable, though, since the VS toolchain does not support writing absolute paths to dependencies in output files. Editing the result files by hand is also a bit suspicious because there are many things that depend on offsets inside the file. Adding or removing even one byte will probably break something. The only thing we can really do is to replace one string with a different one with the same length.

This turns out to be the same problem that rpath entries have on Unix and the solution is also the same. We need to get a long enough string inside the output file and then we can replace it with a different string. If the replacement string is shorter, it can be padded with null bytes because the strings are treated as C strings. I have written a simple test repository doing this, which can be downloaded from Github.

On unix rpath is specified with a command line argument so it can be padded to arbitrary size. Windows does not support this so we need to fake it. The basic idea is simple. Instead of creating a library helper.dll we create a temporary library called aaaaaaaaaaaaaaaaaaaaaaaa.dll and link the program against that. When viewed in a hex editor the executable looks like this.


Now we can copy the library to its real name in a subdirectory and patch the executable. The result looks like this.


The final name was shorter than what we reserved so there are a bunch of zero bytes in the executable. This program can now be run and it will always resolve to the library that we specified. When the program is installed the entry can be changed to just plain helper.dll in the same way making it indistinguishable from libraries built without this trick (apart from the few extra null bytes).

Rpath on Windows: achieved.

Is this practical?

It's hard to say. I have not tested this on anything except toy programs but it does seem to work. It's unclear if this was the intended behaviour, but Microsoft does take backwards compatibility fairly seriously so one would expect it to keep working. The bigger problem is that the VS toolchain creates many other files, such as pdb debug info files, that probably don't like being renamed like this. These files are mostly undocumented so it's difficult to estimate how much work it would take to make binary hotpatching work reliably.

The best solution would be for Microsoft to add a new linker argument to their toolchain that would write dependency info to the files as absolute paths and to provide a program to rewrite those entries as discussed above. Apple already provides all of this functionality in their core toolchain. It would be nice for MS to do the same. This would simplify cross platform development because it would make all the major platforms behave in the same way.

keskiviikko 8. toukokuuta 2019

Why crowdfunding freely licensed documentation is illegal in Finland

On the Meson manual crowdfunding page it is mentioned that the end result can not be put under a fully free license. Several people have said that they "don't believe such a law could exist" or words to that effect. This blog post is an attempt to to explain the issue in English as all available text about the case is in Finnish. As a disclaimer: I'm not a lawyer, the following is not legal advice, there is no guarantee, even that any of the information below is factual.

To get started we need to go back in time a fair bit and look at disaster relief funds. In Finland you must obtain a permit from the police in order to gather money for general charitable causes. This permit has strict requirements. The idea is that you can't just start a fundraising, take people's money and pocket it, instead the money must provably go to the cause it was raised for. The way the law is written is that a donation to charity is done without getting "something tangible" in return. Roughly if you give someone money and get a physical item in return, it is considered a sales transaction. If you give money to someone and in return get a general feeling of making the world better in some way, that is considered a donation. The former is governed by laws of commerce, the latter by laws of charity fundraising.

A few years ago there was a project to create a book to teach people Swedish. The project is page is here, but it is all in Finnish so it's probably not useful to most readers. They had a crowdfunding project to finish the project with all the usual perks. One of the goals of the crowdfunding was to make the book freely distributable after publishing. This is not unlike funding feature work on FOSS projects works.

What happened next is that the police stepped in and declared this illegal (news story, in Finnish). Their interpretation was that participating in this campaign without getting something tangible in return (i.e. paying less than the amount needed to get the book) was a "charitable donation". Thus it needs a charity permit as explained above. Running a crowdfunding campaign is still legal if it is strictly about pre-sales. That is, every person buys "something" and that something needs to have "independent value" of some sort. If the outcome of a project is a PDF and that PDF becomes freely available, it can be argued that people who participated did not get any "tangible value" in exchange for their money.

Because of this the outcome of the Meson manual crowdfunding campaign can not be made freely available. This may seem a bit stupid, but sadly that's the law. The law is undergoing changes (see here, in Finnish), but those changes will not take effect for quite some time and even when they do it is unclear how those changes would affect these kinds of projects.

keskiviikko 1. toukokuuta 2019

The Meson manual crowdfunding campaign

The Meson Build system has existed for over six years. Even though we have a fairly good set of documentation, there has not been a standalone user's manual for it. Until now.


A crowdfunding campaign to finance the manual has just been launched on Indiegogo. The basic deal is simple, for 30€ you get the final book as a PDF. To minimize work and save trees, there is no physical version. There are also no stickers, beer mats or any other tchotchkies. There are a few purchase options as well as opportunities for corporate sponsorships. Please see the Indiegogo project page for further details. If there are any questions about this campaign feel free to contact me. The easiest way is via email.

Overall I'm quite excited about this campaign. One reason is obviously personal, but the other has to do with sustainability of FOSS projects in general. There has been a lot of talk about how maintainers of open source projects can get compensated for their work. This campaign can be seen as an experiment to see if the crowdfunding model could work in practice.

So if you are just getting started with building software and want a user manual, buy this book. If you have basic experience with Meson and want to dive deeper, buy this book. If you are a seasoned veteran and don't really need a book but want to support the project (specifically me), buy this book. Regardless of anything else, please spread the word on your favourite social media and real world venues of choice.

Let the experiment begin!