Full Marks from The Economist

Full Marks from The Economist

The World Ahead 2023 issue of The Economist revealing some of the Superforecasters' forecastsGood Judgment’s team of Superforecasters received full marks from The Economist for their forecasts published last year in “The World Ahead 2023” issue. Now that eight of the nine questions have resolved, The Economist’s editors were able to score the Superforecasters’ performance.

“The Good Judgment team had a good year in 2023, correctly forecasting the outcomes of the eight questions that were resolved,” the editorial team writes in the “The World Ahead 2024” print issue. “Global growth was 3%, China grew by 5%, ruling-party candidates won in Nigeria and Turkey, Vladimir Putin was not ousted, there was no election in Britain, no clash over Taiwan, and no nuclear device detonated by Russia.”

As to the ninth question in the 2023 publication, the Superforecasters continue to see a protracted conflict in Ukraine, likely going beyond 1 October 2024. That question remains open, and, as The Economist team notes, “Events in 2023 did not prove them wrong.”

“The World Ahead 2024” from The Economist is now available online and in print, and once again features the Superforecasters’ take on key questions for 2024. See their forecasts in the newspaper—or subscribe to FutureFirst™ to access all their forecasts that are updated daily.

About Good Judgment

Good Judgment Inc is the successor to the Good Judgment Project, a research team that emerged as the undisputed victor in a massive forecasting competition (the Aggregate Contingent Estimation or ACE tournament) sponsored by the Intelligence Advanced Research Project Activity (IARPA) of the US Office of the Director of National Intelligence (ODNI). Spanning four years, 500 questions, and over a million forecasts, that research project confirmed and refined methods that lead to the best possible forecast accuracy and is credited with the discovery of Superforecasters—people who are exceptionally skilled at assigning accurate probabilities to future outcomes. Good Judgment Inc is now making this winning approach to harnessing the wisdom of the crowd available for commercial use. Good Judgment’s Superforecasters are men and women around the world who go through a rigorous qualification process to demonstrate consistently high accuracy and quality commentary in their forecasting approaches.

About FutureFirst™

FutureFirst, Good Judgment’s exclusive monitoring tool, gives 24/7 access to timely insights on top-of-mind questions from a diverse global team of professional Superforecasters. It combines the advantages of an expert network with model-friendly quantitative forecasts of likely outcomes of key events. Daily forecast updates from our subscription service allow clients to spot emerging risks earlier and see ahead of the competition.

Superforecasters’ Toolbox: Fermi-ization in Forecasting

Superforecasters’ Toolbox: Fermi-ization in Forecasting

Although usually a very private person, Superforecaster Peter Stamp agreed to be interviewed by a major Polish daily, Rzeczpospolita, on Good Judgment’s request. The reporter started the interview with a pop quiz. He asked Peter to estimate the number of tram cars that serve the city of Warsaw, Poland’s capital. Without using the internet, or having ever been to Warsaw, in under three minutes Peter came up with a remarkably accurate answer (only 10% away from the actual number, according to the reporter, Marek Wierciszewski). All he needed to know for his calculations were the typical size of a Warsaw tram and the relative importance of this means of transportation.

The method Peter used was Fermi-ization, and it is one of the key techniques Superforecasters employ to tackle complex questions even with minimal information.

What Is Fermi-ization?

In his day, physicist Enrico Fermi (1901-1954) was known not only for his groundbreaking contributions to nuclear physics. He was also able to come up with surprisingly accurate estimates using scarce information. The technique he used was elegant in its simplicity: He would break down grand, seemingly intractable questions into smaller sub-questions or components that could be analyzed or researched. He would then make educated guesses about each component until he arrived at his final estimate.

Many science and engineering faculties today teach this method, including through assignments like “estimate the number of square inches of pizza the students will eat during one semester.” Instead of blurting out a random number, students are expected to break the question down into smaller bits and engage with each one to produce a thoughtful answer (in this example, the estimate would depend on such factors as the number of students, the number of pizzas a student would eat per week, and the size of an average pizza).

Fermi-ization is a valuable tool in a Superforecaster’s toolbox. Since the days of the original Good Judgment Project and continuing in Good Judgment Inc’s work today, Superforecasters have proved the usefulness of this technique in producing accurate forecasts on seemingly impossible questions—from the scale of bird-flu epidemics, oil prices, and interest rates to election outcomes, regional conflict, and vaccinations during the Covid-19 pandemic.

Uses of Fermi-ization in Forecasting

In their seminal book Superforecasting, Philip Tetlock and Dan Gardner list Fermi-ization as the second of the Ten Commandments for Aspiring Superforecasters. This placement is not a coincidence. In the world of Superforecasters—experts known for their consistently accurate forecasts—Fermi-ization is a fundamental tool, enabling them to arrive at accurate predictions even in response to questions that initially seem impossible to quantify.

“Channel the playful but disciplined spirit of Enrico Fermi,” Tetlock and Gardner write. “Decompose the problem into its knowable and unknowable parts. Flush ignorance into the open. Expose and examine your assumptions. Dare to be wrong by making your best guesses. Better to discover errors quickly than to hide them behind vague verbiage.”

Depending on the question, this process can take just a few minutes, as it did when Peter worked out an estimated number of Warsaw’s tram cars, or it could be methodical, slow, and painstaking. But it is an invaluable road map whether accuracy is the goal.

Fermi-ization in forecasting has multiple uses:

    • It helps the forecaster to avoid the classic cognitive trap of relying on quick-and-easy—and often incorrect!—answers where more thought is called for.
    • It forces the forecaster to sort the relevant components from the irrelevant ones.
    • It enables the forecaster to separate the elements of the question that are knowable from those that are unknowable.
    • It makes the forecasters examine their assumptions more carefully and pushes them toward making educated—rather than blind—guesses.
    • It informs both the outside and the inside view in approaching the question.


Three Steps in Fermi-ization

Fermi-ization becomes easier and increasingly effective with practice. Keep these three steps in mind as you give it a try.

    1. Unpack the question by asking, “What would it take for the answer to be yes? What would it take for it to be no?” or “What information would allow me to answer the question?”
    2. Give each scenario your best estimate.
    3. Dare to be wrong.


Not the Only Tool

Of course, Fermi-ization is not the only tool in a Superforecaster’s toolbox. Mitigation of cognitive biases, ability to recognize and minimize noise, being actively open-minded, and keeping scores are all crucial components of the Superforecasting process. You can learn these techniques during one of our Superforecasting Workshops, or you can pose your own questions for Superforecasters to engage with through a subscription to FutureFirst™.