The human brain comprehends atoms but it does not comprehend electrons

The most common thing people know about software projects is that they fail. They might get delayed for months and years. They might be ten times more expensive than expected. They might be completely unusable pieces of garbage. And so on. Over the past 50 or so years many theories have been presented on why that is and how things could be made better. A lot of progress has been made, but no fundamental breakthrough has been made. Software projects still fail fairly often for unexpected reasons.

Typically this implies that there is some fundamental issue (or, more likely, several of them) causing these problems. Since nobody really knows what the real cause is, we are free to wildly speculate and hypothesize that the problem lies somewhere in the human brain. Not any single person's brain, mind you, but fundamental properties of the brain itself.

What the brain does incredibly well

The human brain is an incredible piece of work. Millions of years of evolution has adapted it to be able to be able to do astonishing things in the blink of an eye. The fact that you can walk and talk at the same time without needing to use any conscious effort on it is nothing short of amazing. As a concrete example, look at the following picture for a couple of seconds and then continue reading.

Just a quick glimpse should give you a fairly good idea about the quality of these two very different vehicles. These include things like top speed, acceleration, insurance costs, ride comfort, perceived increase in sex appeal, fuel consumption, maintenance effort, reliability, how long they are expected to keep working and so on. The estimates you get are probably not 100% accurate, but altogether they are probably mostly correct. Given your current life situation you probably also immediately "know" which one of these two would be better for you or that neither is suitable and that you need something completely different, such as a van or a bicycle.

This is your brain doing the thing it does best: examining physical objects in the real world. This works because things that are made of atoms have noticeable and persistent properties. These include things like mass, durability, sharpness and so on. This consistency allows one to fairly accurately estimate their behaviour even in unexpected situations.

And where it stumbles

If you ever find yourself bored at a party, try the following. Find some people, explain the Monty Hall Problem to them and give the correct answer. Then spend the rest of the evening watching how people will argue until they are blue in the face that you are wrong and that the probability is 50/50 and changing the door is not beneficial. This will happen even if (and especially when) participants have higher education degrees in STEM fields.

The Monty Hall Problem is a great example of the deficiencies of our brain. The actual problem is simple but the solution is so counter-intuitive that many people will refuse to accept it and will instead spend massive amounts of time and energy to disprove the point (in vain) because it just "does not feel right". In mathematics this happens fairly often. The reason this is relevant to the discussion at hand is that, at the end, that is what all software really is: applied mathematics. It does not obey the rules of the physical world and atoms, it only exists in the realm of logic. Computers don't deal with atoms, only electrons[1]. To see how this translates to coding, let's do a thought experiment.

Suppose your job is to paint a wall. After a hard day of painting you have ten meters of freshly painted wall. The next day you get to work and continue painting. At the end of the day you look at your work: exactly ten meters wall covered in paint. Puzzled you check everything but can't find anything wrong. You should have 20 meters of painted wall but don't. The next day the same thing happens again. This makes your manager unhappy so he appoints three people to help. The next day the four of you start painting, working as hard and as effectively as you possibly can. When the day comes to an end, you have seven meters of painted wall. Somehow three meters of paint you had already spplied have disappeared and nobody can explain why or even where the paint went. Determined to make up for the loss everyone works extra hard and at the end of the day, finally, nine meters of the wall is properly covered in paint. Everyone goes home exhausted but happy to have finally made progress. The next morning it is discovered that both the paint and the wall it was applied to have disappeared.

Finally, software

This is the essence of creating software. Since it exists purely in the realm of logic and mathematics, you can't really see it, smell it, touch it or feel it. It has failure modes unfathomable with physical processes and these problems occur fairly regularly. This means that all the builtin functionality of your brain is useless in evaluating the outcome and quality of a software project. Code can only experienced through thinking, which is slow, difficult and error prone. As opposed to cars, comparing two different code bases for quality and suitability is a big undertaking. But it gets worse.

The output of code, like the web browser you are using to read this blog post, look and feel a lot like physical objects made of atoms. This means that when people who have no personal experience in programming either buy or manage software projects, they are going to do it as if they were dealing with physical real world objects. That is what they are trained to do and have years of experience in after all. It is also actively harmful, because software is electrons and, as such, beyond the immediate instictive grasp of the human brain. It does not play by the rules of atoms and trying to make it do so will only lead to failure. 

[1] For the physicists among you, yes, technically this should be electric fields, not electrons. This is an artistic decision to make the headline more clickbaity.

Short term usability is not the same as long term usability

When designing almost any piece of code or other functionality, the issue of usability often comes up. Most of the time these discussions go well, but at other times fairly big schisms appear. It may even be that two people advocate for exactly opposite approaches, both claiming that their approach is better because of "usability". This seems like a paradox but upon closer examination we find that it's not for the simple reason that there is no one thing called usability. What is usable or not depends on a ton of different things.

Short term: maximal flexibility and power

As an example let's imagine a software tool that does something. What exactly it does is not important, but let's assume that it needs to be configurable and customizable. A classical approach to this is to create a small core and then add a configuration language for end users. The language may be Turing complete either by design or it might grow into Turing completeness accidentally when people add new functionality as needed. Most commonly this happens via some macro expansion functionality added to "reduce boilerplate".

Perhaps the best known example of this kind of tool is Make. At its core it is only a DAG solver and you can have arbitrary functionality just by writing shell script fragments direcly inside your Makefile. The underlying idea is to give developers the tools they need to solve their own special problems. These kinds of accidental programming environments are surprisingly common. I was told that a certain Large Software Company that provides services over the web did a survey on how many self created Turing complete configuration languages they have within their company. I never managed to get an actual number out of them, which should tell you that the real answer is probably "way too many".

Still, Make has stood the test of time as have many other projects using this approach. Like everything in life this architecture also has its downsides. The main one can be split in two parts, the first of which is this:

Even though this approach is easy to get started and you can make it do anything (that is what Turing completness does after all), it fairly quickly hits a performance ceiling. Changing the configuration and adding new features becomes ever more tedious as the size of the project grows. This is known as the Turing tarpit, where everything is possible but nothing is easy.

In practice this seems to happen quite often. Some would say every time. To understand why, we need to remember Jussi's law of programmers:

The problem with programmers is that if you give them the chance, they will start programming.

An alternative formulation would be that if you give programmers the possibility to solve their own problems, they are going to solve the hell out of their own problems. These sorts of architectures play the strings of the classical Unix hacker's soul. There are no stupid limitations, instead the programmer is in charge and can fine tune the solution to match the exact specifics and details of their unique snowflake project. The fact that the environment was not originally designed for general programming is not a deterrent, but a badge of merit. Making software do things it was not originally designed to do is what the original hacker ethic was all about, after all. This means that every project ends up solving the same common problems in a different way. This is a violation of the DRY principle over multiple projects. Even though every individual project only solves any problem once, globally they get solved dozens or even hundreds of times. The worst part is that none of these solutions work together. Taking to arbitrary Make projects and trying to build one as a subproject of another will not work and will only lead to tears.

Long term: maximal cooperation

An alternative approach is to create some sort of a configuration interface which is specific to the problem at hand but prevents end users from doing arbitrary operations. It looks like this.

The main thing to note is that at the beginning this approach is objectively worse. If you start using one of these at the beginning it is entirely possible that it can't solve your problem, and you can't make it work on your own, you need to change the system (typically by writing and sending pull requests). This brings enormous pressure to use an existing tweakable system because even though it might be "worse" at least it works.

However once this approach gets out of its Death Valley and has support for the majority of things users need, then networks effects kick in and everything changes. It turns out that perceived snowflakeness is very different from actual snowflakeness. Most problems are in fact pretty much the same for most projects and the solutions for those are mostly boring. Once there is a ready made solution, people are quite happy to use that, meaning that any experience you have in one project is immediately transferable to all other projects using the same tool. Additionally as general programming has been prevented, you can be fairly sure [1] that no cowboy coder has gone and reimplemented basic functionality just because they know better or, more probably, have not read the documentation and realized that the functionality they want already exists.

[1] Not 100% sure, of course, because cowboy coders can be surprisingly persistant.