Superforecasters: Still Crème de la Crème Six Years On

Written by

Superforecasters: Still Crème de la Crème Six Years On

The multi-year geopolitical forecasting tournament sponsored by the research arm of the US Intelligence Community (IARPA) that led to the groundbreaking discovery of “Superforecasters” ended in 2015. Since then, public and private forecasting platforms and wisdom-of-the-crowd techniques have only proliferated. Six years on, are Good Judgment’s Superforecasters still more accurate than a group of regular forecasters? What, if anything, sets their forecasts apart from the forecasts of a large crowd?

A bar graph showing the Superforecasters' error scores are lower than those of regular forecasters
From the paper: Superforecasters’ accuracy outstrips wisdom-of-the-crowd scores.

A new white paper by Dr. Chris Karvetski, senior data and decision scientist with Good Judgment Inc (GJ Inc), compares six years’ worth of forecasts on the GJ Inc Superforecaster platform and the GJ Open public forecasting platform to answer these questions.

Key takeaway: Superforecasters, while a comparatively small group, are significantly more accurate than their GJ Open forecasting peers. The analysis shows they can forecast outcomes 300 days prior to resolution better than their peers do at 30 days from resolution.

Who are “Superforecasters”?

During the IARPA tournament, Superforecasters routinely placed in the top 2% of accuracy among their peers and were a winning component of the experimental research program of the Good Judgment Project, one of five teams that competed in the initial tournaments. Notably, these elite forecasters were over 30% more accurate than US intelligence analysts forecasting the same events with access to classified information.

Key Findings

Calibration plot showing the Superforecasters are 79% closer to perfect calibration
From the paper: Regular forecasters tend to show overconfidence, whereas the Superforecasters are close to perfect calibration.

Dr. Karvetski’s analysis presented in “Superforecasters: A Decade of Stochastic Dominance” uses forecasting data over a six-year period (2015-2021) on 108 geopolitical forecasting questions that were posted simultaneously on Good Judgment Inc’s Superforecaster platform (available to FutureFirst™ clients) as well as the Good Judgment Open (GJ Open) forecasting platform, an online forecasting platform that allows anyone to sign up, make forecasts, and track their accuracy over time and against their peers.

The data showed:

  • Despite being relatively small in number, the Superforecasters are much more prolific, and make almost four times more forecasts per question versus GJ Open forecasters.
  • They are also much more likely to update their beliefs via small, incremental changes to their forecast.
  • Based on the Superforecasters’ daily average error scores, they are 35.9% more accurate than their GJ Open counterparts.
  • Aggregation has a notably larger effect on GJ Open forecasters; yet, the Superforecaster aggregate forecasts are, on average, 25.1% more accurate than the aggregate forecasts using GJ Open forecasts.
  • The average error score for GJ Open forecasters at 30 days from resolution is larger than any of the average error scores of Superforecasters on any day up to 300 days prior to resolution.
  • GJ Open forecasters, in general, were over-confident in their forecasts. The Superforecasters, in contrast, are 79% better calibrated. “This implies a forecast from Superforecasters can be taken at its probabilistic face value,” Dr. Karvetski explains.
  • Finally, the amount of between-forecaster noise is minimal, implying the Superforecasters are better at translating the variety of different signals into a numeric estimate of chance.

You can read the full paper here.

Where Can I Learn More About Superforecasting?

Subscription to FutureFirst, Good Judgment’s exclusive monitoring tool, gives clients 24/7 access to Superforecasters’ forecasts to help companies and organizations quantify risk, improve judgment, and make better decisions about future events.

Our Superforecasting workshops incorporate Good Judgment research findings and practical Superforecaster know-how. Learn more about private workshops, tailored to the needs of your organization, or public workshops that we offer.

A journey to becoming a Superforecaster begins at GJ Open. Learn more about how to become a Superforecaster.

Books on Making Better Decisions

Written by

Books on Making Better Decisions: Good Judgment’s Back-to-School Edition

Since the publication of Tetlock and Gardner’s seminal Superforecasting: The Art and Science of Prediction, many books and articles have been written about the ground-breaking findings of the Good Judgment Project, its corporate successor Good Judgment Inc, and the Superforecasters.

This is not surprising: Decision-makers have a lot to learn from the Superforecasters. Thanks to being actively open-minded and unafraid to rethink their conclusions, the Superforecasters have been able to make accurate predictions where experts often failed. They know how to think in probabilities (or “in bets”), reduce the noise in their judgments, and mitigate cognitive biases such as overconfidence. As Tetlock and Good Judgment Inc have shown, these are skills that can be learned.

Here is a short list of eight notable books that present a wealth of information on ways to evaluate an uncertain future and improve decision-making.

In 2011, IARPA—the research arm of the US intelligence community—launched a massive competition to identify cutting-edge methods to forecast geopolitical events. Four years, 500 questions, and over a million forecasts later, the Good Judgment Project (GJP)—led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania—emerged as the undisputed victor in the tournament. GJP’s forecasts were so accurate that they even outperformed those of intelligence analysts with access to classified data. One of the biggest discoveries of GJP were the Superforecasters: GJP research found compelling evidence that some people are exceptionally skilled at assigning realistic probabilities to possible outcomes—even on topics outside their primary subject-matter training.

In their New York Times bestseller, Superforecasting, our cofounder Philip Tetlock and his colleague Dan Gardner profile several of these talented forecasters, describing the attributes they share, including open-minded thinking, and argue that forecasting is a skill to be cultivated, rather than an inborn aptitude.

Noise, defined as unwanted variability in judgments, can be corrosive to decision-making. Yet, unlike its better-known companion, bias, it often remains undetected—and therefore unmitigated—in decision processes. In addition to research-based insights into better decision-making and remedies to identify and reduce noise as a source of error, Kahneman and his colleagues take a close look at a select group of forecasters—the  Superforecasters—whose judgments are not only less biased but also less noisy than those of most decision-makers. As co-author of Noise Cass Sunstein says, “Superforecasters are less noisy—they don’t show the variability that the rest of us show. They’re very smart; but also, very importantly, they don’t think in terms of ‘yes’ or ‘no’ but in terms of probability.”

Intelligence is often seen as the ability to think and learn, but in a rapidly changing world, there’s another set of cognitive skills that might matter more: the ability to rethink and unlearn. As an organizational psychologist, Adam Grant investigates how we can embrace the joy of being wrong, bring nuance to charged conversations, and build schools, workplaces, and communities of lifelong learners. He also profiles Good Judgment Inc’s Superforecasters Kjirste Morrell and Jean-Pierre Beugoms, who embody the outstanding thought processes suggested in the book. You can read more about Morrell and Beugoms in our interviews here.

David Epstein examines the world’s most successful athletes, artists, musicians, inventors, and forecasters to show that in most fields—especially those that are complex and unpredictable—generalists, not specialists, are primed to excel. In a chapter about the failure of expert predictions, he discusses Phil Tetlock’s research, the GJP, and how “a small group of foxiest forecasters—just bright people with wide-ranging interests and reading habits—destroyed the competition” in the IARPA tournament. Good Judgment Inc’s Superforecasters Scott Eastman and Ellen Cousins, profiled in the book, weigh in on such topics as curiosity, aggregating perspectives, and learning from specialists without being swayed by their often narrow worldviews.

Other books that mention Superforecasting, Good Judgment Inc, or Good Judgment Project

Super Quiet

Written by

Super Quiet: Kahneman’s Noise and the Superforecasters

Much is written about the detrimental role of bias in human judgment. Its companion, noise, on the other hand, often goes undetected or underestimated. Noise: A Flaw in Human Judgment, the new book by Nobel laureate Daniel Kahneman and his co-authors, Olivier Sibony and Cass R. Sunstein, exposes how noise—variability in judgments that should be identical—wreaks havoc in many fields, from law to medicine to economic forecasting.

Noise offers research-based insights into better decision-making and suggests remedies to reduce the titular source of error.

No book on making better judgments, of course, particularly better judgments in forecasting, would be complete without the mention of Superforecasters, and certainly not one co-authored by such a luminary of human judgment as Kahneman.

Superforecasters (discussed in detail in chapters 18 and 21 of the book) are a select group who “reliably out-predict their less-than-super peers” because they are able to consistently overcome both bias and noise. One could say, the Superforecasters are not only actively open-minded—they are also super quiet in their forecasts.

“What makes the Superforecasters so good?” the authors ask. For one, they are “unusually intelligent” and “unusually good with numbers.” But that’s not it.

“Their real advantage,” according to Kahneman, Sibony, and Sunstein, “is not their talent at math; it is their ease in thinking analytically and probabilistically.”

Noise identifies other qualities that set the Superforecasters apart from regular forecasters:

    • Willingness and ability to structure and disaggregate problems;
    • Taking the outside view;
    • Systematically looking for base rates.

In short, it’s not just their natural intelligence. It’s how they use it.

Not everyone is a good forecaster, of course, and while crowds are usually better than individuals, not every crowd is equally wise.

“It is obvious that in any task that requires judgment, some people will perform better than others will. Even a wisdom-of-crowds aggregate of judgments is likely to be better if the crowd is composed of more able people,” the authors state.

Good Judgment’s Superforecasters are unique, with an unbeaten track record, among a myriad of individual forecasters and forecasting firms. Kahneman, Sibony, and Sunstein are not surprised:

“Judgments are both less noisy and less biased when those who make them are well trained, are more intelligent, and have the right cognitive style.”

Good Judgment’s Training Reduces Noise

“Well trained” is a key word here. When the Superforecasters were discovered in “some of the most innovative work on the quality of forecasting”—the Good Judgment Project (GJP, 2011-2016)—they were the top 2% among thousands of volunteers. That doesn’t mean, however, that the rest of the world is doomed to drown in noisy decision-making. It is not an either-you-have-it-or-you-don’t skill.

According to Kahneman, Sibony, and Sunstein, “people can be trained to be superforecasters or at least to perform more like them.”

Good Judgment Inc’s online training and workshops do just that. Based on the concepts taught in the GJP training, these workshops are designed to reduce psychological biases—which, in turn, results in less noise.

Kahneman and colleagues explain how this works, citing the BIN (bias, information, and noise) model for forecasting developed by Ville Satopää, Marat Salikhov, and Good Judgment’s co-founders Phil Tetlock and Barb Mellers:

“When they affect different individuals on different judgments in different ways, psychological biases produce noise. … As a result, training forecasters to fight their psychological biases works—by reducing noise.”

Good Judgment’s training also focuses on teaming, another effective method scientifically demonstrated to reduce noise.

According to Kahneman, Sibony, and Sunstein, both private and public organizations—and the society at large—stand to gain much from reducing noise. “Should they do so, organizations could reduce widespread unfairness—and reduce costs in many areas,” the authors write. And the Superforecasters are an example for decision-makers to emulate in these efforts.

Learn Superforecasting from the pros at our Superforecasting Workshops
See upcoming workshops here