The Rationality Gap

We’re good at solving problems, but not as good at seeing how other people solve theirs

Why did so many expect the Millennium Bug to cause a disaster?

Why was the peak oil theory so widely believed?

Why is rent control so popular?

Because we suffer from the sleepwalking illusion: we keep missing how people adapt to changing circumstances. Many failed to see that programmers would fix the Millennium Bug, or that oil companies would increase production through new extraction techniques. And rent control supporters overlook that developers respond by building less. We often see other people as more static than they really are. As I will show in a series of posts, this illusion distorts our entire worldview.

I’ll start by explaining the illusion’s psychological roots: the rationality gap between actors and observers. As actors, we usually respond fairly rationally to new situations. But as observers of other people, we often fail to see this. We don’t live up to the same standards.

One reason is that it’s hard to put yourself in the shoes of someone trying to solve a problem like the Millennium Bug. As the situation unfolds, they often discover new possibilities that outside observers can’t imagine. While actors think in concrete near mode – where the world is full of details – observers think in abstract far mode, where most of these details are lost.

The other reason is an asymmetry of attention. Actors typically focus on a single issue they really need to get right. The programmers who fixed the Millennium Bug knew they could be held personally responsible for failure. By contrast, observers typically track many events and pay a much smaller price if they get some of them wrong. The Millennium Bug doomsayers could move on with little consequence.

Everyday rationality

The rationality gap is pervasive in ordinary life. Joining a fleeing crowd on a station platform is often viewed as a textbook case of human irrationality. But other people running is actually evidence that something is wrong. People who might be in danger grasp this logic, while outside observers often dismiss it as senseless panic.

Another example is advice columnists telling people to stop hiding stress at work. In fact, it’s often sensible to keep quiet, since many firms don’t want to promote people who risk getting overwhelmed. But while employees are rightly worried about this, advice columnists have much less at stake and often choose the warm glow of recommending courage.

Or consider the debate about credentials. People often scoff at the focus on prestigious degrees, but there is a rationale. In the absence of more reliable information, they’re an imperfect but useful signal of competence. Recruiters know this, and job applicants know that they know it. But many onlookers don’t bother to think it through, instead pushing the populist story of vain status-seeking.

Unfortunately, psychologists have lent support to this overly pessimistic view of human decision-making. Part of the problem is academic incentives, as journals have long favoured findings of bias and error. Many of these findings were overturned in the replication crisis, but public discourse has been slow to catch up. The idea that we’re largely irrational lives on.

Institutional rationality

If we underestimate other people, we underestimate our institutions even more. When we discuss society, we often fall into a cynical mood and don’t even try to describe it fairly. For instance, many people dismiss politicians as much less competent than ordinary people, contrary to what the evidence suggests. I think this dismissal arises from that cynical mood, not from a serious assessment of the facts.

The media compounds this tendency by focusing disproportionately on institutional failures. While disasters that actually occur dominate the news, those that institutions prevent are all but forgotten. Over time, this gives a badly skewed picture of how well institutions work.

Another factor is that institutions are typically harder to understand than individuals. Many of them are complex and opaque, handling technical questions beyond most observers’ grasp. Our pessimistic biases face little resistance, leaving many people with a level of cynicism that’s completely unmoored from reality.

This isn’t to say that institutions don’t make mistakes – of course they do. But on average, they’re not as incompetent as we intuitively think. If we zoom out and take a broader perspective, we see that Western societies work remarkably well. I’ll soon return to this theme.