A Strong Year for Good Judgment Inc: 2022 in Review and Outlook

Written by

A Strong Year for Good Judgment Inc: 2022 in Review and Outlook

What a year! With Russia’s invasion of Ukraine, protests in Iran, rising tensions in the Taiwan Strait, and inflation pressures around the world, 2022 was the busiest year yet at Good Judgment. Our esteemed clients and FutureFirst™ subscribers in the private sector, government, and non-profit organizations posed a record 181 questions to the professional Superforecasters. More than 80 questions resolved and were scored in 2022. We launched an additional 422 questions on Good Judgment Open, our public forecasting site and primary training ground for future Superforecasters.

Good Judgment’s Superforecasters’ outlook on the 2022 US midterm elections as of 22 October 2021, published in The Economist.

At the start of 2022, our Superforecasters called the 2022 US midterm election, as can be seen in The Economist’s The World Ahead publication, an ongoing collaboration that showcases how “data-driven approaches are becoming popular in all kinds of journalism.” Other appearances of Good Judgment and Superforecasting in the press and news can be found here. The Superforecasters also nailed the forecasts on Jerome Powell’s renomination to head the Federal Reserve and on the Tokyo Olympics. Looking ahead, their forecasts focus on the Russia-Ukraine conflict, tensions around Taiwan, global economy, and key elections in 2023 (many of which are featured in The World Ahead 2023).

Here are some of the other key developments and projects we worked on in the past year:

We launched an updated version of our subscription-based forecast monitor, FutureFirst. In addition to a brand-new interface, the monitor has been equipped with the following features:

    • Forecast Channels: Questions are grouped by theme or topical focus. Some of the current channels are US Elections, Ukraine, Geopolitics, Economics, Policy, Markets, China, and the Federal Reserve. These are available as part of FutureFirst or as standalone subscriptions.
    • Implied Medians: In addition to probability ranges, for specific questions FutureFirst now includes a median as part of a continuous probability distribution.
    • API Access: FutureFirst REST API enables clients’ programs to retrieve daily updated forecast data for use in their models. Separate entry points deliver Question data, Forecast data, and Forecast Distribution data (for applicable questions). Data can be provided in JSON or CSV formats.
Good Judgment’s Superforecasters called Jerome Powell’s renomination to lead the Fed months in advance.

We have further refined our question cluster methodology to illuminate complex topics with a diverse set of discrete questions which themselves can be valuable but taken together provide decision makers with a robust real-time monitoring tool.

We expanded our workshops to include advanced training sessions on question generation and low probability/high impact “gray swan” events. We resumed offering in-person workshops in 2022 even as we continued to offer virtual trainings for teams scattered across the globe—with an average Net Promoter Score of 71 across all sessions, above Apple. Of all the organizations that had a workshop in 2021, more than 90% came back for more in 2022, now regularly sending their interns and new hires through our training.

The US military continues to lead the way: Superforecasting training has been part of the curriculum for senior officers since 2019, and Good Judgment has been delivering semester-long courses at National Defense University in Washington, DC, continuously since the start of 2020. Over the same period, civil servants and military officers from Dubai have been forecasting during and after our annual two week-training courses.

We launched the Forecasting Aptitude Survey Tool, a screening tool that measures characteristics that Good Judgment’s research found correlate with subsequent forecasting accuracy. It’s been an integral part of our workshops for years, and we’re now pleased to provide it to organizations looking for an additional input to evaluate their existing or prospective staff.

To help advance decision-making skills among high school students, Good Judgment partnered with the Alliance for Decision Education to launch a pilot forecasting challenge for selected schools across the United States. We are continuing and expanding this project in 2023. At the other end of the education spectrum, we collaborated with the Kennedy School of Government at Harvard University in a semester-long course that honed the forecasting skills of top graduate students.

The Superforecasters’ forecast on a peace agreement between Russia and Ukraine.

With climate change becoming a growing concern worldwide, we partnered with adelphi, a leading European think tank for climate, environment, and development, to produce “Seven questions for the G7: Superforecasting climate-fragility risks for the coming decade,” published in 2022. Commissioned by the multilaterally backed climate security initiative Weathering Risk, it is the first report of its kind that applies the Superforecasting methodology to climate-related risks to peace and stability. We also partnered with Dr. William MacAskill, the author of the groundbreaking 2022 book What We Owe the Future, supplying 22 additional forecasts on the long-term impacts of climate change.

Finally, at the end of 2022, Good Judgment’s co-founder Philip Tetlock launched the non-profit Forecasting Research Institute, an exciting additional venture that will advance the frontiers of forecasting for better decisions. We look forward to contributing to their efforts in 2023—and beyond.

It’s already shaping up to be another exciting year of uncertainty in 2023, with gridlock in Washington, rising geopolitical tensions, and fierce global economic headwinds. We look forward to those challenges and contributing our forecasts and insights with the help of a phenomenal staff at Good Judgment Inc, the unrivalled skills of our professional Superforecasters, and the support of our expanding roster of clients and partners around the world.

Please sign up for our newsletter to keep up with our news in 2023.

Good Judgment Inc and Metaculus Launch First Collaboration

Written by

Good Judgment Inc and Metaculus Launch First Collaboration

Metaculus and Good Judgment Inc are pleased to announce our first collaboration. Our organizations, which represent two of the largest human judgment forecasting communities in the world, will compare our results and methodologies in a project comprised of identical forecasting questions that ask about the future of 10 Our World In Data metrics. We plan to share insights, lessons learned, and analysis to contribute to the broader community and to the science of forecasting.

Cohorts of Superforecasters from Good Judgment Inc and Pro Forecasters from Metaculus will make predictions on their separate platforms on a set of 10 questions about technological advances, global development, and social progress on time horizons ranging from one to 100 years.

A Future Fund grant is supporting both organizations in producing these expert forecasts, as well as a public tournament on the Metaculus platform, though this collaboration between Metaculus and GJI is distinct, separate, and voluntary.

“Our shared goal is advancing forecasting as a trusted method for leaders to make critical decisions,” said Gaia Dempsey, CEO of Metaculus. “We’re thrilled to be working with our partners at Good Judgment Inc. This is the first time two of the largest players in the field of forecasting have come together in the spirit of collaboration to compare methodologies and to advance the science of forecasting.”

“We’re excited to be partnering with Metaculus to combine our approaches to apply probabilistic thinking to an uncertain future and help individuals and organizations make better decisions about the future,” said Warren Hatch, Good Judgment’s CEO. “We look forward to building on this collaboration for Our World In Data.”

Good Judgment Inc harnesses the wisdom of the crowd, led by Superforecasters, to quantify hard-to-measure risks for smarter strategic decisions for the private and public sectors.

Metaculus is a forecasting technology platform that optimally aggregates quantitative predictions of future events.

How Distinct Is a “Distinct Possibility”?

Written by

How Distinct Is a “Distinct Possibility”?
Vague Verbiage in Forecasting

“What does a ‘fair chance’ mean?”

It is a question posed to a diverse group of professionals—financial advisers, political analysts, investors, journalists—during one of Good Judgment Inc’s virtual workshops. The participants have joined the session from North America, the EU, and the Middle East. They are about to get intensive hands-on training to become better forecasters. Good Judgment’s Senior Vice President Marc Koehler, a Superforecaster and former diplomat, leads the workshop. He takes the participants back to 1961. The young President John F. Kennedy asks his Joint Chiefs of Staff whether a CIA plan to topple the Castro government in Cuba would be successful. They tell the president the plan has a “fair chance” of success.

The workshop participants are now asked to enter a value between 0 and 100—what do they think is the probability of success of a “fair chance”?

When they compare their numbers, the results are striking. Their answers range from 15% to 75% with the median value of 60%.

Figure 1. Meanings behind vague verbiage according to a Good Judgment poll. Source: Good Judgment.

The story of the 1961 Bay of Pigs invasion is recounted in Good Judgment co-founder Philip Tetlock’s Superforecasting: The Art and Science of Prediction (co-authored with Dan Gardner). The advisor who wrote the words “fair chance,” the story goes, later said what he had in mind was only a 25% chance of success. But like many of the participants in the Good Judgment workshop some 60 years later, President Kennedy took the phrase to imply a more positive assessment of success. By using vague verbiage instead of precise probabilities, the analysts failed to communicate their true evaluation to the president. The rest is history: The Bay of Pigs plan he approved ended in failure and loss of life.

Vague verbiage is pernicious in multiple ways.

1. Language is open to interpretations. Numbers are not.

According to research published in the Journal of Experimental Psychology, “maybe” ranges from 22% to 89%, meaning radically different things to different people under different circumstances. Survey research by Good Judgment shows the implied ranges for other vague terms, with “distinct possibility” ranging from 21% to 84%. Yet, “distinct possibility” was the phrase used by White House National Security Adviser Jake Sullivan on the eve of the Russian invasion in Ukraine.

Figure 2. How people interpret probabilistic words. Source: Andrew Mauboussin and Michael J. Mauboussin in Harvard Business Review.

Other researchers have found equally dramatic perceptions of probability that people attach to vague terms. In a survey of 1,700 respondents, Andrew Mauboussin and Michael J. Mauboussin found, for instance, that the probability range that most people attribute to an event with a “real possibility” of happening spans about 20% to 80%.

2. Language avoids accountability. Numbers embrace it.

Pundits and media personalities often use such words as “may” and “could” without even attempting to define them because these words give them infinite flexibility to claim credit when something happens (“I told you it could happen”) and to dodge blame when it does not (“I merely said it could happen”).

“I can confidently forecast that the Earth may be attacked by aliens tomorrow,” Tetlock writes. “And if it isn’t? I’m not wrong. Every ‘may’ is accompanied by an asterisk and the words ‘or may not’ are buried in the fine print.”

Those who use numbers, on the other hand, contribute to better decision-making.

“If you give me a precise number,” Koehler explains in the workshop, “I’ll know what you mean, you’ll know what you mean, and then the decision-maker will be able to decide whether or not to proceed with the plan.”

Tetlock agrees. “Vague expectations about indefinite futures are not helpful,” he writes. “Fuzzy thinking can never be proven wrong.”

If we are serious about making informed decisions about the future, we need to stop hiding behind hedge words of dubious value.

3. Language can’t provide feedback to demonstrate a track record. Numbers can.

In some fields, the transition away from vague verbiage is already happening. In sports, coaches use probability to understand the strengths and weaknesses of a particular team or player. In weather forecasting, the standard is to use numbers. We are much better informed by “30% chance of showers” than by “slight chance of showers.” Furthermore, since weather forecasters get ample feedback, they are exceptionally well calibrated: When they say there’s a 30% chance of showers, there will be showers three times out of ten—and no showers the other seven times. They are able to achieve that level of accuracy by using numbers—and we know what they mean by those numbers.

Another well-calibrated group of forecasters are the Superforecasters at Good Judgment Inc, an international team of highly accurate forecasters selected for their track record among hundreds and hundreds of others. When assessing questions about geopolitics or the economy, the Superforecasters use numeric probabilities that they update regularly, much like weather forecasters do. This involves mental discipline, Koehler says. When forecasters are forced to translate terms like “serious possibility” or “fair chance” into numbers, they have to think carefully about how they are thinking, to question their assumptions, and to seek out arguments that can prove them wrong. And their track record is available for all to see. All this leads to better informed and accurate forecasts that decision-makers can rely on.

 

Good Judgment Inc is the successor to the Good Judgment Project, which won a massive US government-sponsored geopolitical forecasting tournament and generated forecasts that were 30% more accurate than those produced by intelligence community analysts with access to classified information. The Superforecasters are still hard at work providing probabilistic forecasts along with detailed commentary and reporting to clients around the world. For more information on how you can access FutureFirst™, Good Judgment’s exclusive forecast monitoring tool, visit https://goodjudgment.com/services/futurefirst/.