Comparing C, C++ and D performance with a real world project

Some time ago I wrote a blog post comparing the real world performance of C and C++ by converting Pkg-config from C to C++ and measuring the resulting binaries. This time we ported it to D and running the same tests.

Some caveats

I got comments that the C++ port was not "idiomatic C++". This is a valid argument but also kind of the point of the test. It aimed to test the behavior of ported code, not greenfield rewrites. This D version is even more unidiomatic, mostly because this is the first non-trivial D project I have ever done. An experienced D developer could probably do many of the things much better than what is there currently. In fact, there are parts of the code I would do differently based solely on the things I learned as the project progressed.

The code is available in this Github repo. If you wish to use something else than GDC, you probably need to tweak the compiler flags a bit. It also does not pass the full test suite. Once the code was in good enough condition to pass the Gtk+ test needed to get the results on this post, motivation to keep working on it dropped a fair bit.

The results

The result array is the same as in the original post, but the values for C++ using stdlibc++ have been replaced with corresponding measurements from GDC.

                                    GDC   C++ libc++       C

Optimized exe size                364kB        153kB    47kB

minsize exe size                  452kB        141kB    43kB

3rd party dep size                    0            0   1.5MB

compile time                       3.9s         3.3s    0.1s

run time                          0.10s       0.005s  0.004s

lines of code                      3249         3385    3388

memory allocations                  151         8571    5549

Explicit deallocation calls           0            0      79

memory leaks                          7            0   >1000

peak memory consumption            48.8kB         53kB    56kB

Here we see that code size is not D's strong suite. As an extra bit of strangeness the size optimized binary took noticeably more space than the regular one.  Compile times are also unexpectedly long given that D is generally known for its fast compile times. During development GDC felt really snappy, though, printing error messages on invalid code almost immediately. This would indicate that the slowdown is coming from GDC's optimization and code generation passes.

The code base is the smallest of the three but not by a huge margin. D's execution time is the largest of the three but most of that is probably due to runtime setup costs, which are amplified in a small program like this.

Memory consumption is where things get interesting. D uses a garbage collector by default whereas C and C++ don't, requiring explicit deallocation either manually or with RAII instead. The difference is clear in the number of allocations done by each language. Both C and C++ have allocation counts in the thousands whereas D only does 151 of them. Even more amazingly it manages to beat the competition by using the least amount of memory of any of the tested languages.

Memory graphs

A massif graph for the C++ program looked like this:

This looks like a typical manual memory management graph with steadily increasing memory consumption until the program is finished with its task and shuts down. In comparison D looks like the following:

D's usage of a garbage collector is readily apparent here. It allocates a big chunk up front and keeps using it until the end of the program. In this particular case we see that the original chunk was big enough for the whole workload so it did not need to grow the size of the memory pool. The small jitter in memory consumption is probably due to things such as file IO and work memory needed by the runtime.

The conversion and D as a language

The original blog posted mentioned that converting the C program to C++ was straightforward because you could change things in very small steps (including individual items in structs) while keeping the entire test suite running for the entire time. The D conversion was the exact opposite.

It started from the C++ one and once the files were renamed to D, nothing worked until all of the code was proper D. This meant staring at compiler failure messages and fixing issues until they went away (which took several weeks of work every now and then when free time presented itself) and then fixing all of the bugs that were introduced by the fixes. A person proficient in D could probably have done the whole thing from scratch in a fraction of the time.

As a language D is a slightly weird experience. Parts of it are really nice such as the way it does arrays and dictionaries. Much of it feels like a fast, typed version of Python, but other things are less ergonomic. For example you can do if(item in con) for dictionaries but not for arrays (presumably due to the potential for O(n) iterations).

Perhaps the biggest stepping stone is the documentation. There are nice beginner tutorials  but intermediate level documentation seems to be scarce or possibly it's just hard to bing for. The reference documentation seems to be written by experts for other experts as tersely as possible. For comparison Python's reference documentation is both thorough and accessible in comparison. Similarly the IDE situation is unoptimal, as there are no IDEs in Ubuntu repositories and the Eclipse one I used was no longer maintained and fairly buggy (any programming environment that does not have one button go-to-definition and reliably working ctrl+space is DOA, sorry).

Overall though once you get D running it is nice. Do try it out if you haven't done so yet.