This article is taken from the June 2024 issue of The Critic. To get the full magazine why not subscribe? Right now we’re offering five issues for just £10.

Daniel Kahneman died on 27 March at the age of 90. He was one of the most perceptive and accurate psychologists of the last 100 years, and his analysis of the sorts of mistakes we are liable to make when trying to decide what to do is permanently valuable.

Kahneman’s death was the cue for reverential obituaries outlining his conclusions: we’re very bad at estimating probabilities accurately; we are reluctant to believe things which do not fit in with our existing convictions and with our interests; our decisions are frequently determined by factors that should be irrelevant, such as feeling hungry, or experiencing a cool breeze on a hot day; and we ignore relevant statistics because we’re convinced that we’re exceptional: the statistics might apply to others — they don’t apply to us.

One feature of Kahneman’s achievement that the obituaries did not mention is the profoundly pessimistic view of the social world that it reflects. “Our comforting conviction that the world makes sense,” he wrote in Thinking, Fast and Slow, “rests on a very secure foundation: our almost unlimited ability to ignore our own ignorance” — a sentiment he shared with that great pessimist, Arthur Schopenhauer.

We think we know what we’re doing. But mostly, we don’t. The difference between success and failure in almost every aspect of life is primarily due to luck, and only marginally down to our own skill or merit. An individual’s intentions have very little to do with how his or her life works out. The social world is dominated by accidents and chance. “The good end happily, the bad unluckily. That is what fiction means,” as Miss Prism says in Oscar Wilde’s The Importance of Being Ernest. Kahneman agreed. He said that his two favourite equations from social science were first: success = talent + luck. And second: great success = a little more talent and a lot of luck.

Kahneman thought that one basic error is responsible for most of the mistakes we characteristically make when deciding what to do: optimism. We constantly think that the world is a more benign place than it actually is — and that we understand more about the way it works than we do. He thought most experts were absurdly optimistic about their abilities to predict accurately what would happen when presented with a case in their own area of expertise.

He was particularly struck by the statistics showing that experienced professional judgement is significantly less reliable than algorithms based on very simple rules. Marriage guidance counsellors, for example, who interview married couples and assess the strengths and weaknesses of their relationship, are less reliable at predicting the survival of a marriage than the simple algorithm: the stability of a marriage is proportional to the frequency of lovemaking minus the frequency of arguments. If the result of that calculation produces a negative number, the marriage is unlikely to last — whatever the experts predict.

It is optimism about their own powers, as well as a reluctance to admit their limitations, that leads experts in every field to deny that their judgement is as unreliable as the statistics show it is. Optimism is what leads politicians to embark on grand infrastructural projects — such as new high-speed railway lines or new government buildings — when a cursory look at the relevant statistics would tell them that these usually end up costing five or ten times the projected budget, whilst delivering significantly fewer benefits than claimed. An accurate cost-benefit analysis, as opposed to one based on an optimistic fantasy, would reveal that the benefits will not come close to outweighing the costs.

Optimism makes people ignore known risks on the basis that those risks won’t materialise in their case. It makes them keep on spending money and time on trying to realise dreams even when there is clear evidence that the effort will be wasted: the strong likelihood is that the project will not succeed. Optimism is what prevents us from being able to assess accurately our chances of being able to realise our hopes.

Kahneman endorses Schopenhauer’s claim that “Hope is the confusion of the desire for something with its probability” — and it is optimism that is the source of hope. It makes us think our achievements are only the result of our own skill, when they are usually the consequence of a large dose of luck. It lands us in a make-believe world, where we stay until reality comes crashing back in, frequently in the form of insolvency.

Kahneman had all those reservations about optimism, and many others. But he also thought that optimism is essential to human life: if it didn’t exist, we would all be worse off, possibly to the extent that our lives would be unbearably bad. He believed that to achieve almost anything of significance, an individual has to ignore the statistics that accurately reflect the high chance of failure. This is true whether an individual is aiming at achieving a scientific breakthrough or something valuable in the arts.

Two-thirds of small businesses in the United States fail after five years. People who start small businesses in America either don’t believe those statistics, or think the statistics don’t apply to them. When asked, 80 per cent of entrepreneurs in the US put their personal chances of failure at 3 out of 10 — that is, they thought their chances of success were twice as high as the true figure. Another survey put Americans’ confidence in their own abilities even higher. It recorded that 33 per cent of Americans starting a new business believe their chances of failure are zero.

Most attempts at achieving anything significant end in failure. But if everyone adjusted what they tried to do to the realistic chances of success, almost no-one would attempt to achieve anything remarkable — and that would leave us all much worse off. It is difficult to see how a capitalist economy could work if everyone was completely rational, in the sense of only investing time and money in projects which either had a better than even chance of coming to fruition, or were assessed to have a potential pay-off so large as to make it worthwhile to take the high risk of losing the investment completely.

Optimism isn’t only beneficial because it provides the engine for economic growth and the impetus behind most scientific discoveries. Optimism also has many benefits to optimists. Optimists are happier, healthier and more resilient than people who evaluate the probabilities of success for their ventures accurately. Because it has so many helpful effects, Kahneman thought optimism was perhaps the most valuable quality an individual could have.

This generates the paradox of rationality: it can be rational to be irrational. You are likely to be better off if you do not live your life according to an accurate assessment of probabilities. Of course, the rationality of unreasonable optimism is a question of degree: the wildly over-optimistic individual whose outlook has no connection to reality is not going to do well.

Even when they fail to realise their chosen projects, optimists who are less deluded than that will still do better than those who have a more rigorously accurate assessment of what is likely to happen. The reason? Optimists deal with failure better. They find it easier to embark on new ventures, and are less likely to be depressed by not achieving what they set out to do.

But just as Kahneman thought most of us were incapable of adjusting our beliefs so as to free them of the errors which often characterise them, so he also thought that most of us aren’t able to change ourselves into optimists if we don’t already have a basically optimistic temperament.

If we’re rational enough to be free of the life-enhancing delusions of optimism, we are not going to be able consciously to make ourselves subject to them. That most of us are incorrigibly irrational turns out to be a blessing rather than a curse. It is another aspect of the paradox of “rational irrationality” that hovers over Kahneman’s discoveries about how to identify irrationality in decision-making. Our failures of rationality are frequently what give us hope. Hope is a very precious commodity. A life without it is not an attractive prospect — and may not be a viable one.

Kahneman won the Nobel Prize for economics, but stressed he was “not an economist. I study the way our minds work, not the way the economy does”. Nevertheless, his work has major implications for economics. Almost all of the models economists use to predict what will happen to the economy, and what should be done to ensure that outcomes are better rather than worse, use and depend on the assumption that humans are rational. Kahneman showed that that assumption is a fundamental mistake. People are not rational: we allow unreasonable considerations to determine how we make decisions.

But Kahneman’s insights have had almost no effect on the way economists operate. By far the majority of academic economists continue to construct their models in the same way they always did. But then their failure to take Kahneman’s insights on board is actually just another example of a variety of irrationality that Kahneman studied: the almost universal tendency to discount anything that damages our pride and doesn’t suit our interests. It is not the least of Kahneman’s achievements to have found a new way to make us aware of that ancient truth.