Full Marks from The Economist

Written by

Full Marks from The Economist

The World Ahead 2023 issue of The Economist revealing some of the Superforecasters' forecastsGood Judgment’s team of Superforecasters received full marks from The Economist for their forecasts published last year in “The World Ahead 2023” issue. Now that eight of the nine questions have resolved, The Economist’s editors were able to score the Superforecasters’ performance.

“The Good Judgment team had a good year in 2023, correctly forecasting the outcomes of the eight questions that were resolved,” the editorial team writes in the “The World Ahead 2024” print issue. “Global growth was 3%, China grew by 5%, ruling-party candidates won in Nigeria and Turkey, Vladimir Putin was not ousted, there was no election in Britain, no clash over Taiwan, and no nuclear device detonated by Russia.”

As to the ninth question in the 2023 publication, the Superforecasters continue to see a protracted conflict in Ukraine, likely going beyond 1 October 2024. That question remains open, and, as The Economist team notes, “Events in 2023 did not prove them wrong.”

“The World Ahead 2024” from The Economist is now available online and in print, and once again features the Superforecasters’ take on key questions for 2024. See their forecasts in the newspaper—or subscribe to FutureFirst™ to access all their forecasts that are updated daily.

About Good Judgment

Good Judgment Inc is the successor to the Good Judgment Project, a research team that emerged as the undisputed victor in a massive forecasting competition (the Aggregate Contingent Estimation or ACE tournament) sponsored by the Intelligence Advanced Research Project Activity (IARPA) of the US Office of the Director of National Intelligence (ODNI). Spanning four years, 500 questions, and over a million forecasts, that research project confirmed and refined methods that lead to the best possible forecast accuracy and is credited with the discovery of Superforecasters—people who are exceptionally skilled at assigning accurate probabilities to future outcomes. Good Judgment Inc is now making this winning approach to harnessing the wisdom of the crowd available for commercial use. Good Judgment’s Superforecasters are men and women around the world who go through a rigorous qualification process to demonstrate consistently high accuracy and quality commentary in their forecasting approaches.

About FutureFirst™

FutureFirst, Good Judgment’s exclusive monitoring tool, gives 24/7 access to timely insights on top-of-mind questions from a diverse global team of professional Superforecasters. It combines the advantages of an expert network with model-friendly quantitative forecasts of likely outcomes of key events. Daily forecast updates from our subscription service allow clients to spot emerging risks earlier and see ahead of the competition.

Superforecasters’ Toolbox: Fermi-ization in Forecasting

Written by

Superforecasters’ Toolbox: Fermi-ization in Forecasting

Although usually a very private person, Superforecaster Peter Stamp agreed to be interviewed by a major Polish daily, Rzeczpospolita, on Good Judgment’s request. The reporter started the interview with a pop quiz. He asked Peter to estimate the number of tram cars that serve the city of Warsaw, Poland’s capital. Without using the internet, or having ever been to Warsaw, in under three minutes Peter came up with a remarkably accurate answer (only 10% away from the actual number, according to the reporter, Marek Wierciszewski). All he needed to know for his calculations were the typical size of a Warsaw tram and the relative importance of this means of transportation.

The method Peter used was Fermi-ization, and it is one of the key techniques Superforecasters employ to tackle complex questions even with minimal information.

What Is Fermi-ization?

In his day, physicist Enrico Fermi (1901-1954) was known not only for his groundbreaking contributions to nuclear physics. He was also able to come up with surprisingly accurate estimates using scarce information. The technique he used was elegant in its simplicity: He would break down grand, seemingly intractable questions into smaller sub-questions or components that could be analyzed or researched. He would then make educated guesses about each component until he arrived at his final estimate.

Many science and engineering faculties today teach this method, including through assignments like “estimate the number of square inches of pizza the students will eat during one semester.” Instead of blurting out a random number, students are expected to break the question down into smaller bits and engage with each one to produce a thoughtful answer (in this example, the estimate would depend on such factors as the number of students, the number of pizzas a student would eat per week, and the size of an average pizza).

Fermi-ization is a valuable tool in a Superforecaster’s toolbox. Since the days of the original Good Judgment Project and continuing in Good Judgment Inc’s work today, Superforecasters have proved the usefulness of this technique in producing accurate forecasts on seemingly impossible questions—from the scale of bird-flu epidemics, oil prices, and interest rates to election outcomes, regional conflict, and vaccinations during the Covid-19 pandemic.

Uses of Fermi-ization in Forecasting

In their seminal book Superforecasting, Philip Tetlock and Dan Gardner list Fermi-ization as the second of the Ten Commandments for Aspiring Superforecasters. This placement is not a coincidence. In the world of Superforecasters—experts known for their consistently accurate forecasts—Fermi-ization is a fundamental tool, enabling them to arrive at accurate predictions even in response to questions that initially seem impossible to quantify.

“Channel the playful but disciplined spirit of Enrico Fermi,” Tetlock and Gardner write. “Decompose the problem into its knowable and unknowable parts. Flush ignorance into the open. Expose and examine your assumptions. Dare to be wrong by making your best guesses. Better to discover errors quickly than to hide them behind vague verbiage.”

Depending on the question, this process can take just a few minutes, as it did when Peter worked out an estimated number of Warsaw’s tram cars, or it could be methodical, slow, and painstaking. But it is an invaluable road map whether accuracy is the goal.

Fermi-ization in forecasting has multiple uses:

    • It helps the forecaster to avoid the classic cognitive trap of relying on quick-and-easy—and often incorrect!—answers where more thought is called for.
    • It forces the forecaster to sort the relevant components from the irrelevant ones.
    • It enables the forecaster to separate the elements of the question that are knowable from those that are unknowable.
    • It makes the forecasters examine their assumptions more carefully and pushes them toward making educated—rather than blind—guesses.
    • It informs both the outside and the inside view in approaching the question.


Three Steps in Fermi-ization

Fermi-ization becomes easier and increasingly effective with practice. Keep these three steps in mind as you give it a try.

    1. Unpack the question by asking, “What would it take for the answer to be yes? What would it take for it to be no?” or “What information would allow me to answer the question?”
    2. Give each scenario your best estimate.
    3. Dare to be wrong.


Not the Only Tool

Of course, Fermi-ization is not the only tool in a Superforecaster’s toolbox. Mitigation of cognitive biases, ability to recognize and minimize noise, being actively open-minded, and keeping scores are all crucial components of the Superforecasting process. You can learn these techniques during one of our Superforecasting Workshops, or you can pose your own questions for Superforecasters to engage with through a subscription to FutureFirst™.

Why Young People Should Learn Forecasting

Written by

Why Young People Should Learn Forecasting

It is Good Judgment’s belief that teaching forecasting skills to young people will lead to better life outcomes for all.

Every decision is a forecast. Some are seemingly simple, like leaving home in time to catch the bus. Others are more impactful, like one’s career choice. At no point in life do we need to make so many critical decisions as in our late teens and early twenties. Yet, many young people are under-equipped in their approaches to making decisions.

Research in neuroscience and psychology shows that the prefrontal cortex—a region of the brain associated with reasoning and decision-making—does not mature until well into a person’s twenties. For this reason, many young people tend to overlook long-term consequences, discount larger delayed rewards over smaller immediate gains, and succumb to peer influence and undervalue risk. This leads to suboptimal choices, some of which may have repercussions for the individual’s well-being for years to come.

At Good Judgment, we believe that equipping young people with forecasting skills will help improve their overall decision-making abilities and lead to better outcomes.

Five Reasons Why Young People Should Learn Forecasting

  1. Forecasting teaches students how to predict future trends based on historical data. This helps inform their decisions in many fields, not least personal finance and career choices.
  2. Forecasting teaches students to recognize and minimize cognitive biases, such as overconfidence (for instance, failure to recognize gaps in one’s knowledge) and scope insensitivity (incorrect assessment of the scope of a problem or opportunity). Failure to account for cognitive biases in decision-making often leads to inferior choices and undervalued risks.
  3. Forecasting helps students grasp the concept of uncertainty in real-world scenarios. It prepares them to make educated guesses, even when complete information is unavailable.
  4. Forecasting teaches critical thinking. To make accurate forecasts, students must learn to analyze data, discern patterns, and consider multiple factors that could influence outcomes. The skills that come with critical thinking are valuable in many areas of life and work.
  5. Forecasting prepares students for a fast-changing world. We live in a world of rapid technological advancements, socio-economic shifts, and environmental changes. Young people who understand forecasting are better able to anticipate these changes and face these shifts proactively and with an open mind.

Good Judgment has long upheld a vision of forecasting becoming part and parcel of education. This vision is shared by many of our colleagues and friends as well as partners at education institutions across North America, Europe, and beyond. Since the inception of Good Judgment Inc, it has been an honor and pleasure to be part of many worthy programs that aim to bring forecasting into the lecture halls and classrooms—from a partnership with the Alliance for Decision Education in a pilot project for high schools in the United States, to hosting a GJ Open Challenge for Harvard Kennedy School students, to workshops for the best and brightest at the University of Copenhagen (Denmark) and elsewhere.

As the number of forecasting courses continues to grow, we look to a future of better decision-making, whether in personal lives, business, or policymaking.

If you represent an institution that shares our goal of promoting better decision-making among young people, we can support you by:

  • Designing and conducting training, from short workshops to semester-long courses;
  • Hosting GJ Open Challenges for your students to practice forecasting;
  • Providing forecaster feedback reports prepared by our data science team;
  • Providing mentoring by Superforecasters; and more.

If you are an individual interested in improving your forecasting acumen, join the internet’s smartest crowd on GJ Open, follow the advice from our Superforecasters (here, here, or here), read good books (some lists here and here), or explore such options as our self-paced online training course. And above all, keep practicing and keep track of your progress. As Phil Tetlock and Dan Gardner write in Superforecasting, “Forecasters who practice get better at distinguishing finer degrees of uncertainty, just as artists get better at distinguishing subtler shades of gray.”