A Closer Look at a Superforecaster’s
Scientific, Data-based Method

Mechanical engineer and Superforecaster® Kjirste Morrell is featured as an exemplar of outstanding thought processes in best-selling author and top Wharton professor Adam Grant’s latest book, Think Again. Below, she shares insights into her data-driven process for reviewing questions and determining her forecast.

Let me use the question on the number of Federal firearms background checks in April through June 2021. This question is based on a specific set of data that is influenced by many potential factors. The topic evokes a strong emotional response which can bring biases into the forecast. Focus on the data first and make sure you’re looking at the right data.

As with most questions that are about a number from a specific site, the very first step is to go to that site and look at the historical data. Get the correct data into a spreadsheet, using whatever method works best for you. Start making plots, such as the number of monthly background checks by year and over time, to see if there is a seasonal trend and to see if there is a general trend.

Even looking at the table of numbers, a few things are obvious, so there might be a temptation to skip plotting. I think plotting is worthwhile here, and the two plots I’ve attached emphasize different aspects for this particular set of data.  A few of the things that I might wonder about with any data collected over several years are whether there is a seasonal trend or if a general trend is apparent. In this case, there are some seasonal effects: there’s a peak in December, with a secondary peak in March, usually. The number of background checks also rises over time with increasing variability, finishing with 2020 and early 2021’s much higher and more variable numbers.

The first question I have is what monthly averages correspond to each bin in the question and how those compare to historical data. Lines at those averages have been added to the figures. In January there were 4.3 M checks, which is the largest monthly amount so far. February was dramatically lower. Similarly, the average of 2.67 M is below any month in the last year. I can imagine events that lead to the total for April-June being either fewer than 8 M or more than 14 M, but some bins may be less likely than others.

Going forward, each month I would check the data at the FBI site and add in a new data point. Consider questions like: Does April rule out another bin or indicate a trend? What are the new monthly averages that would need to be met to end up in each bin?

Once I have a reasonable sense of what the historical data looks like, I like to make a list of factors that could impact the number in question. A few that occur to me here are:

• Are any new laws going into effect that will require more background checks?
• Is reported violence rising or falling?
• Does it seem like fear of future violence/unrest is rising or falling?
• Has gun control legislation been discussed recently in news outlets?
• Could supply affect this number?
• Are there other limiting factors, like max throughput or number of people?
• What is behind the seasonal peaks, especially March—are there sales events that explain that?
• Do I understand where the number of background checks comes from?

Understanding what the data represents is especially worthwhile, and there is more information at the FBI site about the background check system. (Starting here: https://www.fbi.gov/services/cjis/nics.) Some of the other reports and statistics may be useful, perhaps something there is a leading indicator, more fine-grained data, or suggests another way of looking at the information.

That’s the process I would go through. Probably I would only get partway on the first pass and then add more when revisiting the question later.

Summary:

• Go to the FBI link. The most important thing is to know what the data looks like for the source that will be used to settle the question.
• Download the FBI data and put it into a spreadsheet.
• Graph yearly and over time
• Break the bin boundaries into average per month and plot with the data
  • 8 M is an average of 2.67 M/month,
  • 10 M is an average of 3.33 M/month,
  • 12 M is an average of 4 M/month,
  • 14 M is an average of 4.67 M/month
• How reasonable is it that any of these averages will be met in April-June 2021?
• As data is added for April & May, does that rule out any bins?
• Make a list of factors that might affect the data and investigate those.

How would you forecast this question on Good Judgment Open?

Learn to think like a Superforecaster with our Online Training course!
The three short modules provide a solid foundation for novice forecasters in fields such as finance, strategy, and consulting.

Forecasting the Pandemic: Dealing with Death and Other Emotional Subjects

More than two million people are dead worldwide, with one-fourth of them in the United States. Such are the grim milestones the world has reached in its struggle with COVID-19.

These numbers came as little surprise to professional Superforecasters, who have been making accurate forecasts on these topics up to a year in advance.

“Unprecedented”

The world surpassed two million deaths attributed to COVID-19 on 15 January 2021. The US death toll reached 500,000 people on 22 February.

A year ago, in March 2020, the unfolding crisis was marked by extreme uncertainty and was widely described as “unprecedented.” Yet, Superforecasters have been assigning the highest probabilities to correct outcomes for worldwide cases and deaths since March 2020, and for US cases and death toll since June 2020.

As the pandemic progressed, Good Judgment forecasts proved “eerily accurate,” as Time magazine put it. “We’re on track to hit two million in reported deaths worldwide if things remain as they are,” one Superforecaster wrote on 26 August. “Add in seasonality, and things will get worse.”

And that’s exactly what happened, as we now know, almost five months after that forecast was made.

“Unless the pace starts to slacken dramatically, all signs point to around 525,000 or thereabouts before the end of March,” another Superforecaster wrote on the US death toll question in January 2021.

Forecasting Emotionally Difficult Questions

Of course, the fact that the COVID-19 numbers did not come as a surprise to Superforecasters as a group hardly makes working on such questions any easier.

“Forecasting deaths is difficult emotionally,” one of Good Judgment’s top Superforecasters, Kjirste Morrell, shared in an interview last month. “Rationally, I know my forecast has no effect on the number of deaths, but I still feel bad, slightly guilty, saying how many people I think will die.”

For emotionally difficult questions, less adept forecasters frequently fall into a cognitive trap called the “social desirability bias.”

In short, the social desirability bias is a cognitive defense mechanism that, when faced with emotionally painful questions, downplays the negatives—from under-reporting less favorable behavior in surveys to underestimating the extent of catastrophic events in forecasting.

The implications could be vast: from invalidating research findings to underestimating the urgency or extent of required response to major events, such as pandemics, conflicts, or migration crises.

Professional Superforecasters are trained to recognize cognitive biases early and mitigate them through multiple effective strategies.

“Earth as an Ant Farm”

Ryan Adler, one of Good Judgment’s Directors and a Superforecaster himself, has described his strategy when forecasting on emotionally wrenching topics: “I try to think of Earth as an ant farm, with me on the outside looking in. From that perspective, it’s easier to forecast solely on what I think will happen, being influenced as little as possible by what I may hope will happen. India car sales is not a brutal topic. Boko Haram’s reign of terror in West Africa, however, is.”

Marc Koehler, Good Judgment’s Senior Vice President and also a Superforecaster, teaches a similar strategy in workshops the company offers to decision-makers in government, finance, and education institutions: “I try to adopt the point of view of a Martian anthropologist, examining homo sapiens through my telescope.”

“The most difficult part of being a professional Superforecaster is not confusing your preferred outcome with the most likely outcome,” says Jean-Pierre Beugoms, another leading Good Judgment Superforecaster interviewed last month. “The task of leaving out one’s biases is even harder when the most likely outcome is a looming disaster. When that happens, I approach the forecast with the following mindset: Forewarned is forearmed.”

To keep cognitive biases in check, Beugoms writes in advance a list of factors that could make him change his predictions.

“Failing to anticipate disasters has done society much harm.”

Superforecaster Jean-Pierre Beugoms

“I also try to approach these unpleasant questions with a sense of mission,” he adds. “Failing to anticipate disasters has done society much harm. An influential Superforecaster, with a record of correctly predicting disasters while avoiding false alarms, may well help society avoid future disasters or, if this proves impossible, can help it prepare for the recovery.”

Morrell has a similar view: “I remind myself that grim forecasts might result in a different course of action being taken. I can hope that changed conditions will make my forecast incorrect.”

What’s next for the world in the struggle against the COVID-19 pandemic? We continue to forecast the number of cases and deaths from COVID-19 and the rollout of vaccinations worldwide. We have delivered accurate forecasts about the timing of vaccine distribution around the world. Newer questions ask about herd immunity in the US and globally as well as how the new normal will look for “work from home” trends, office vacancy rates, and airport capacity. See the latest from Superforecasters on our Public Dashboard or through our exclusive all-access monitor, FutureFirst™.

---

Early Views on the Pandemic