Below, we present the full forecasting history, along with the Superforecasters’ Brier scores, for some of the questions that have appeared on our public dashboard. The Brier score, in simple terms, measures how far a prediction falls from the eventual outcome on a 0 to 2 scale. The smaller the score the more accurate the forecast, with 0 representing perfect prescience.
Below, we present the full forecasting history for some of those questions, along with the Superforecasters’ Brier scores. Check out our public dashboard for publicly available questions or learn about our new subscription dashboard, FutureFirst, to see the Superforecasters’ ongoing predictions on the progress of COVID-19 vaccines, US and European politics, geopolitical risks, and US and global economic and financial indicators.
The COVID-19 pandemic halted nearly all air travel, with airports around the world seeing a precipitous drop at the height of the outbreak. We asked Superforecasters to assess how London Area airport traffic for January 2021 will compare with January 2020. By July 2020, Superforecasters were already placing the highest probability on the eventual outcome, saying the number would be lower by more than 60%. The Brier score for their aggregate forecast was 0.08 (on a 0-2 scale, where 0 reflects perfect prescience).
As Superforecasters were tracking numbers and trends, they were bracing for the worst regarding the highest seven-day median of COVID-19-related confirmed new cases in the United States. As one Superforecaster noted back in November, “Should Good Judgment add a Bin F?” The question closed in Bin E (“More than 250,000”), the highest-probability option for Superforecasters except for brief swings in November. The Brier score for our aggregate forecast was 0.05 (on a 0-2 scale, where 0 reflects perfect prescience).
The question closed on 2 February 2021, resolving in the timeframe “Between 21 January 2021 and 31 May 2021.” Superforecasters have been predicting this outcome since the launch of this question except for about a month when optimistic trends suggested the question could resolve even sooner. Missed distribution goals in late December led Superforecasters to revert to the earlier forecast and expect resolution in early February. The Brier score for our aggregate forecast was 0.11 (on a 0-2 scale, where 0 reflects perfect prescience).
US retail sales have had dramatic swings during the COVID-19 pandemic, placing the fate of retailers’ all-important holiday season in flux. However, thanks to online shopping, US holiday season sales beat what Bloomberg called “low expectations” for the pandemic year. Superforecasters saw it coming, and saw it early, achieving a Brier score of 0.16, where 0 means perfect prescience.
Superforecasters approached the question of the highest seven-day median of COVID-19 confirmed new cases in WHO’s Europe Region with initial optimism. However, a closer look at the numbers and trends sent the most pessimistic answer option, “More than 59,163,” on an upward trajectory. It overtook the other options in September, earning Superforecasters a Brier score on the right side of “maybe” of 0.52 (on a 0-2 scale, where 0 reflects perfect prescience).
Except for a brief period in late August, the Superforecasters were “on the right side of maybe” on this question throughout the forecasting period, converging on near-certainty in the final week. Accordingly, the Brier score for our aggregate forecast was a solid 0.2087.
The FDA’s decision to give Remdesivir full approval as a COVID-19 treatment came more rapidly than more Superforecasters had anticipated. Their commentary had correctly identified Remdesivir as being most likely to be the first COVID treatment to receive full FDA approval; however, they expected the FDA to take more time in reviewing the application, especially because the drug was already available for prescription via Emergency Use Authorization. Being off on timing resulted in a relatively high Brier score of 0.7455 on a 0-2 scale, where 0 reflects perfect prescience.
The Superforecasters correctly anticipated from the day the question was launched that the US would have 200,000 or more confirmed new COVID cases in either Q4 20 or Q1 21. Their confident and accurate forecasts earned an extremely low Brier score of 0.0112 for this question.
This question resolved on September 21st when the New York City public schools resumed limited in-person instruction for pre-K and certain other students. The aggregate Superforecast over time, as displayed on the historical chart below, strongly projected this outcome from the first day the question was open for forecasting and accordingly received an extremely low 0.0068 Brier score.
This question resolved on August 10th when UCLA resumed limited in-person instruction for certain medical-school courses. The aggregate Superforecast over time, as displayed on the historical chart below, received a 0.1244 Brier score.
Good Judgment’s Superforecasters truly knocked this forecast out of the park, achieving a 0.007 Brier score on this question. As with the Magic Kingdom question, they showed early insight by:
The compound forecasting question makes direct comparisons difficult; however, we display other publicly available information on the graph below to provide context for evaluating the Superforecasters’ projections at each point in time.
Footnotes:
Superforecasters achieved a 0.04 Brier score on this forecasting question, showing early insight by:
Their forecasts compared favorably to other publicly available forecasts and statements from Disney management, as displayed on the graph below.
Schedule a consultation to learn how our FutureFirst monitoring tool, custom Superforecasts, and training services can help your organization make better decisions.