Books on Making Better Decisions

Books on Making Better Decisions: Good Judgment’s Back-to-School Edition

Since the publication of Tetlock and Gardner’s seminal Superforecasting: The Art and Science of Prediction, many books and articles have been written about the ground-breaking findings of the Good Judgment Project, its corporate successor Good Judgment Inc, and the Superforecasters.

This is not surprising: Decision-makers have a lot to learn from the Superforecasters. Thanks to being actively open-minded and unafraid to rethink their conclusions, the Superforecasters have been able to make accurate predictions where experts often failed. They know how to think in probabilities (or “in bets”), reduce the noise in their judgments, and mitigate cognitive biases such as overconfidence. As Tetlock and Good Judgment Inc have shown, these are skills that can be learned.

Here is a short list of eight notable books that present a wealth of information on ways to evaluate an uncertain future and improve decision-making.

In 2011, IARPA—the research arm of the US intelligence community—launched a massive competition to identify cutting-edge methods to forecast geopolitical events. Four years, 500 questions, and over a million forecasts later, the Good Judgment Project (GJP)—led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania—emerged as the undisputed victor in the tournament. GJP’s forecasts were so accurate that they even outperformed those of intelligence analysts with access to classified data. One of the biggest discoveries of GJP were the Superforecasters: GJP research found compelling evidence that some people are exceptionally skilled at assigning realistic probabilities to possible outcomes—even on topics outside their primary subject-matter training.

In their New York Times bestseller, Superforecasting, our cofounder Philip Tetlock and his colleague Dan Gardner profile several of these talented forecasters, describing the attributes they share, including open-minded thinking, and argue that forecasting is a skill to be cultivated, rather than an inborn aptitude.

Noise, defined as unwanted variability in judgments, can be corrosive to decision-making. Yet, unlike its better-known companion, bias, it often remains undetected—and therefore unmitigated—in decision processes. In addition to research-based insights into better decision-making and remedies to identify and reduce noise as a source of error, Kahneman and his colleagues take a close look at a select group of forecasters—the  Superforecasters—whose judgments are not only less biased but also less noisy than those of most decision-makers. As co-author of Noise Cass Sunstein says, “Superforecasters are less noisy—they don’t show the variability that the rest of us show. They’re very smart; but also, very importantly, they don’t think in terms of ‘yes’ or ‘no’ but in terms of probability.”

Intelligence is often seen as the ability to think and learn, but in a rapidly changing world, there’s another set of cognitive skills that might matter more: the ability to rethink and unlearn. As an organizational psychologist, Adam Grant investigates how we can embrace the joy of being wrong, bring nuance to charged conversations, and build schools, workplaces, and communities of lifelong learners. He also profiles Good Judgment Inc’s Superforecasters Kjirste Morrell and Jean-Pierre Beugoms, who embody the outstanding thought processes suggested in the book. You can read more about Morrell and Beugoms in our interviews here.

David Epstein examines the world’s most successful athletes, artists, musicians, inventors, and forecasters to show that in most fields—especially those that are complex and unpredictable—generalists, not specialists, are primed to excel. In a chapter about the failure of expert predictions, he discusses Phil Tetlock’s research, the GJP, and how “a small group of foxiest forecasters—just bright people with wide-ranging interests and reading habits—destroyed the competition” in the IARPA tournament. Good Judgment Inc’s Superforecasters Scott Eastman and Ellen Cousins, profiled in the book, weigh in on such topics as curiosity, aggregating perspectives, and learning from specialists without being swayed by their often narrow worldviews.

Other books that mention Superforecasting, Good Judgment Inc, or Good Judgment Project

Super Quiet

Super Quiet: Kahneman’s Noise and the Superforecasters

Much is written about the detrimental role of bias in human judgment. Its companion, noise, on the other hand, often goes undetected or underestimated. Noise: A Flaw in Human Judgment, the new book by Nobel laureate Daniel Kahneman and his co-authors, Olivier Sibony and Cass R. Sunstein, exposes how noise—variability in judgments that should be identical—wreaks havoc in many fields, from law to medicine to economic forecasting.

Noise offers research-based insights into better decision-making and suggests remedies to reduce the titular source of error.

No book on making better judgments, of course, particularly better judgments in forecasting, would be complete without the mention of Superforecasters, and certainly not one co-authored by such a luminary of human judgment as Kahneman.

Superforecasters (discussed in detail in chapters 18 and 21 of the book) are a select group who “reliably out-predict their less-than-super peers” because they are able to consistently overcome both bias and noise. One could say, the Superforecasters are not only actively open-minded—they are also super quiet in their forecasts.

“What makes the Superforecasters so good?” the authors ask. For one, they are “unusually intelligent” and “unusually good with numbers.” But that’s not it.

“Their real advantage,” according to Kahneman, Sibony, and Sunstein, “is not their talent at math; it is their ease in thinking analytically and probabilistically.”

Noise identifies other qualities that set the Superforecasters apart from regular forecasters:

    • Willingness and ability to structure and disaggregate problems;
    • Taking the outside view;
    • Systematically looking for base rates.

In short, it’s not just their natural intelligence. It’s how they use it.

Not everyone is a good forecaster, of course, and while crowds are usually better than individuals, not every crowd is equally wise.

“It is obvious that in any task that requires judgment, some people will perform better than others will. Even a wisdom-of-crowds aggregate of judgments is likely to be better if the crowd is composed of more able people,” the authors state.

Good Judgment’s Superforecasters are unique, with an unbeaten track record, among a myriad of individual forecasters and forecasting firms. Kahneman, Sibony, and Sunstein are not surprised:

“Judgments are both less noisy and less biased when those who make them are well trained, are more intelligent, and have the right cognitive style.”

Good Judgment’s Training Reduces Noise

“Well trained” is a key word here. When the Superforecasters were discovered in “some of the most innovative work on the quality of forecasting”—the Good Judgment Project (GJP, 2011-2016)—they were the top 2% among thousands of volunteers. That doesn’t mean, however, that the rest of the world is doomed to drown in noisy decision-making. It is not an either-you-have-it-or-you-don’t skill.

According to Kahneman, Sibony, and Sunstein, “people can be trained to be superforecasters or at least to perform more like them.”

Good Judgment Inc’s online training and workshops do just that. Based on the concepts taught in the GJP training, these workshops are designed to reduce psychological biases—which, in turn, results in less noise.

Kahneman and colleagues explain how this works, citing the BIN (bias, information, and noise) model for forecasting developed by Ville Satopää, Marat Salikhov, and Good Judgment’s co-founders Phil Tetlock and Barb Mellers:

“When they affect different individuals on different judgments in different ways, psychological biases produce noise. … As a result, training forecasters to fight their psychological biases works—by reducing noise.”

Good Judgment’s training also focuses on teaming, another effective method scientifically demonstrated to reduce noise.

According to Kahneman, Sibony, and Sunstein, both private and public organizations—and the society at large—stand to gain much from reducing noise. “Should they do so, organizations could reduce widespread unfairness—and reduce costs in many areas,” the authors write. And the Superforecasters are an example for decision-makers to emulate in these efforts.

Learn Superforecasting from the pros at our Superforecasting Workshops
See upcoming workshops here