A Closer Look at a Superforecaster’s Scientific, Data-based Method

A Closer Look at a Superforecaster’s
Scientific, Data-based Method

Mechanical engineer and Superforecaster® Kjirste Morrell is featured as an exemplar of outstanding thought processes in best-selling author and top Wharton professor Adam Grant’s latest book, Think Again. Below, she shares insights into her data-driven process for reviewing questions and determining her forecast.

Kjirste Morrell is one of the most accurate Superforecasters

Let me use the question on the number of Federal firearms background checks in April through June 2021. This question is based on a specific set of data that is influenced by many potential factors. The topic evokes a strong emotional response which can bring biases into the forecast. Focus on the data first and make sure you’re looking at the right data.

As with most questions that are about a number from a specific site, the very first step is to go to that site and look at the historical data. Get the correct data into a spreadsheet, using whatever method works best for you. Start making plots, such as the number of monthly background checks by year and over time, to see if there is a seasonal trend and to see if there is a general trend.

Even looking at the table of numbers, a few things are obvious, so there might be a temptation to skip plotting. I think plotting is worthwhile here, and the two plots I’ve attached emphasize different aspects for this particular set of data.  A few of the things that I might wonder about with any data collected over several years are whether there is a seasonal trend or if a general trend is apparent. In this case, there are some seasonal effects: there’s a peak in December, with a secondary peak in March, usually. The number of background checks also rises over time with increasing variability, finishing with 2020 and early 2021’s much higher and more variable numbers.

The first question I have is what monthly averages correspond to each bin in the question and how those compare to historical data. Lines at those averages have been added to the figures. In January there were 4.3 M checks, which is the largest monthly amount so far. February was dramatically lower. Similarly, the average of 2.67 M is below any month in the last year. I can imagine events that lead to the total for April-June being either fewer than 8 M or more than 14 M, but some bins may be less likely than others.

Going forward, each month I would check the data at the FBI site and add in a new data point. Consider questions like: Does April rule out another bin or indicate a trend? What are the new monthly averages that would need to be met to end up in each bin?

Once I have a reasonable sense of what the historical data looks like, I like to make a list of factors that could impact the number in question. A few that occur to me here are:

  • Are any new laws going into effect that will require more background checks?
  • Is reported violence rising or falling?
  • Does it seem like fear of future violence/unrest is rising or falling?
  • Has gun control legislation been discussed recently in news outlets?
  • Could supply affect this number?
  • Are there other limiting factors, like max throughput or number of people?
  • What is behind the seasonal peaks, especially March—are there sales events that explain that?
  • Do I understand where the number of background checks comes from?

Understanding what the data represents is especially worthwhile, and there is more information at the FBI site about the background check system. (Starting here: https://www.fbi.gov/services/cjis/nics.) Some of the other reports and statistics may be useful, perhaps something there is a leading indicator, more fine-grained data, or suggests another way of looking at the information.

That’s the process I would go through. Probably I would only get partway on the first pass and then add more when revisiting the question later.


  • Go to the FBI link. The most important thing is to know what the data looks like for the source that will be used to settle the question.
  • Download the FBI data and put it into a spreadsheet.
  • Graph yearly and over time
  • Break the bin boundaries into average per month and plot with the data
    • 8 M is an average of 2.67 M/month,
    • 10 M is an average of 3.33 M/month,
    • 12 M is an average of 4 M/month,
    • 14 M is an average of 4.67 M/month
  • How reasonable is it that any of these averages will be met in April-June 2021?
  • As data is added for April & May, does that rule out any bins?
  • Make a list of factors that might affect the data and investigate those.

How would you forecast this question on Good Judgment Open?

Learn to think like a Superforecaster with our Online Training course!
The three short modules provide a solid foundation for novice forecasters in fields such as finance, strategy, and consulting.

Forecasting the Pandemic: Dealing with Death and Other Emotional Subjects

Forecasting the Pandemic: Dealing with Death and Other Emotional Subjects

More than two million people are dead worldwide, with one-fourth of them in the United States. Such are the grim milestones the world has reached in its struggle with COVID-19.

These numbers came as little surprise to professional Superforecasters, who have been making accurate forecasts on these topics up to a year in advance.


The world surpassed two million deaths attributed to COVID-19 on 15 January 2021. The US death toll reached 500,000 people on 22 February.

Good Judgment CEO Warren Hatch on CBS News, 22 June 2020

A year ago, in March 2020, the unfolding crisis was marked by extreme uncertainty and was widely described as “unprecedented.” Yet, Superforecasters have been assigning the highest probabilities to correct outcomes for worldwide cases and deaths since March 2020, and for US cases and death toll since June 2020.

As the pandemic progressed, Good Judgment forecasts proved “eerily accurate,” as Time magazine put it. “We’re on track to hit two million in reported deaths worldwide if things remain as they are,” one Superforecaster wrote on 26 August. “Add in seasonality, and things will get worse.”

And that’s exactly what happened, as we now know, almost five months after that forecast was made.

“Unless the pace starts to slacken dramatically, all signs point to around 525,000 or thereabouts before the end of March,” another Superforecaster wrote on the US death toll question in January 2021.

Superforecasters saw the highest likelihood of more than 53 million COVID-19 cases and 800,000 to 8 million COVID-related deaths worldwide a year in advance.

Forecasting Emotionally Difficult Questions

Of course, the fact that the COVID-19 numbers did not come as a surprise to Superforecasters as a group hardly makes working on such questions any easier.

“Forecasting deaths is difficult emotionally,” one of Good Judgment’s top Superforecasters, Kjirste Morrell, shared in an interview last month. “Rationally, I know my forecast has no effect on the number of deaths, but I still feel bad, slightly guilty, saying how many people I think will die.”

According to Superforecasters, the US has been on track to more than 23 million COVID-19 cases and over 350,000 deaths since June 2020.

For emotionally difficult questions, less adept forecasters frequently fall into a cognitive trap called the “social desirability bias.”

In short, the social desirability bias is a cognitive defense mechanism that, when faced with emotionally painful questions, downplays the negatives—from under-reporting less favorable behavior in surveys to underestimating the extent of catastrophic events in forecasting.

The implications could be vast: from invalidating research findings to underestimating the urgency or extent of required response to major events, such as pandemics, conflicts, or migration crises.

Professional Superforecasters are trained to recognize cognitive biases early and mitigate them through multiple effective strategies.

“Earth as an Ant Farm”

Ryan Adler, one of Good Judgment’s Directors and a Superforecaster himself, has described his strategy when forecasting on emotionally wrenching topics: “I try to think of Earth as an ant farm, with me on the outside looking in. From that perspective, it’s easier to forecast solely on what I think will happen, being influenced as little as possible by what I may hope will happen. India car sales is not a brutal topic. Boko Haram’s reign of terror in West Africa, however, is.”

Marc Koehler, Good Judgment’s Senior Vice President and also a Superforecaster, teaches a similar strategy in workshops the company offers to decision-makers in government, finance, and education institutions: “I try to adopt the point of view of a Martian anthropologist, examining homo sapiens through my telescope.”

“The most difficult part of being a professional Superforecaster is not confusing your preferred outcome with the most likely outcome,” says Jean-Pierre Beugoms, another leading Good Judgment Superforecaster interviewed last month. “The task of leaving out one’s biases is even harder when the most likely outcome is a looming disaster. When that happens, I approach the forecast with the following mindset: Forewarned is forearmed.”

To keep cognitive biases in check, Beugoms writes in advance a list of factors that could make him change his predictions.

“Failing to anticipate disasters has done society much harm.”

Superforecaster Jean-Pierre Beugoms

“I also try to approach these unpleasant questions with a sense of mission,” he adds. “Failing to anticipate disasters has done society much harm. An influential Superforecaster, with a record of correctly predicting disasters while avoiding false alarms, may well help society avoid future disasters or, if this proves impossible, can help it prepare for the recovery.”

Morrell has a similar view: “I remind myself that grim forecasts might result in a different course of action being taken. I can hope that changed conditions will make my forecast incorrect.”

What’s next for the world in the struggle against the COVID-19 pandemic? We continue to forecast the number of cases and deaths from COVID-19 and the rollout of vaccinations worldwide. We have delivered accurate forecasts about the timing of vaccine distribution around the world. Newer questions ask about herd immunity in the US and globally as well as how the new normal will look for “work from home” trends, office vacancy rates, and airport capacity. See the latest from Superforecasters on our Public Dashboard or through our exclusive all-access monitor, FutureFirst™.

Early Views on the Pandemic

Officials and Media

CDC Director Robert Redfield, 30 January 2020: “We still believe the immediate risk to the American public is low.”

New Scientist, 11 February 2020: “Could the new coronavirus really kill 50 million people worldwide? […] The short answer is that no one knows.”

USA Today, 17 February 2020: “Fauci doesn’t want people to worry about coronavirus, the danger of which is ‘just minuscule.’”

The Atlantic, 26 March 2020: “Are we winning the war against COVID-19? In the fog of pandemic, we simply don’t know.” 

President Donald Trump, 27 March 2020: “I’m not sure anybody even knows what it is.”

Scientific American, 1 June 2020: “This coronavirus is unprecedented in the combination of its easy transmissibility, a range of symptoms going from none at all to deadly, and the extent that it has disrupted the world.”


Cheddar, 28 May 2020: “92 Percent Chance of U.S. Deaths Exceeding 200,000, Superforecaster Says”

Time, 11 June 2020: “’Superforecasters’ Are Making Eerily Accurate Predictions about COVID-19. Our Leaders Could Learn from Their Approach”

CBS News, 22 June 2020: “Forecasting the COVID-19 Pandemic: Firm Predicts 49% Chance that US Coronavirus-Related Deaths Will Exceed 350,000”

Forbes, 13 August 2020: “Superforecasters Predict Vaccine Next Year, Key to the Economic Forecast”

New Statesman, 20 October 2020: “The ‘Superforecasters’ at Good Judgment [Inc] – an American firm that produces impressively accurate predictions of world affairs – put the chances of an approved vaccine being widely available in the US before April 2021 as high as 70 per cent.”

Forecasting 2021: Superforecasters’ Personal Expectations for the New Year (Part 2)

Forecasting 2021: Superforecasters’ Personal Expectations for the New Year (Part 2)

Clean energy meeting
“In hindsight, everyone will say they saw it coming.”

Turn of the year is traditionally the time when people like to make predictions. Professional Superforecasters do it better and see the future sooner. In celebrating the New Year, the Superforecasters challenged themselves to come up with 2021 predictions in each of three categories:

 Safe But Not Boring (events that are 75% or more likely to happen in 2021)

 Coin Flip (45%-55% likelihood of happening in 2021)

Long Shot (5%-25% likelihood of happening in 2021)

This is Part 2 of the Superforecasters’ personal* expectations for the new year. Here is the link to Part 1.

Real Actions, Not Just Rhetoric: Climate

: Despite extreme weather events, including bush fires and forest fires worldwide, 2021 may be a year of hope — and action — for the environment. The Superforecasters’ slam-dunk forecast on a US return to the Paris Agreement has already been realized. They also predict a new cap on CO2 emissions set in the US and Europe. The Biden climate agenda should involve massive infrastructure investment and serious commitment to carbon-neutral electricity generation by 2035.

“Real actions, not just rhetoric,” one Superforecaster® writes. “In hindsight, everyone will say they saw it coming.”

: Another Superforecaster is also optimistic: “The world will come to a realization that pandemics and climate change can’t be solved outside of a rules-based international system.”


: In geopolitics, some Superforecasters foresee a US return to the Iran Nuclear Agreement…

: …and even small odds for the completion of US withdrawal from Afghanistan without an ensuing major increase in violence.

: The United States embassy to the nation of Israel is expected to be located in Jerusalem.

: Benjamin Netanyahu may have served his last day as prime minister of Israel in 2021, and Israel may normalize relations with at least one country in the MENA region.

: Over in the UK, Boris Johnson should still be PM, while the SNP should win the Scottish Parliament elections.

: Individual Superforecasters also predict failures of authoritarian/populist regimes over their handling of COVID-19…

: …with the possibility of protests or riots in at least one country due to the lack of access to the vaccines.


: Starting with increased diversity in the Biden Cabinet, women should begin to make a dent in male-dominated professions, in both government and business.

“Equal percentages of men and women on boards will begin to be the norm, which will commence in 2021,” one Superforecaster predicts.

Technology: Solving the Last Mile Problem

: Virtual conference apps will continue their ascent in 2021. Starlink will be able to provide Internet connectivity anywhere in the world. The implications of this are less clear:

“This kind of global access is going to change things in ways we haven’t started to imagine yet, largely solving the last mile problem,” a Superforecaster writes.

: Kickstarting the remaking of transportation infrastructure, 2021 may see automated freight routes and customer delivery turn into standard operating practice.

: One Superforecaster goes further: “Amazon or Walmart will have an up-and-running robot delivery force in place to use in remote delivery areas as well as urban areas by Q4 2021.”

: Another Superforecaster gives odds, however small, of Tesla’s move toward full autonomous driving: “No passenger vehicle will be approved by the NHTSA for full autonomous driving AND Tesla Motors will update the software on at least one of its models to allow full autonomous driving.”

: Thanks to technological progress, a new vaccine might be developed for a major disease (e.g., malaria) by the end of 2021.

: On the other hand, this year might also see the first international crisis triggered by the use of deep fake technology.

We Want to Hear from You

Would you like to see Good Judgment track collective forecasts on some of these questions on our Public Dashboard? Let us know by taking part in the poll below.

* This article is based on views of individual Superforecasters, without aggregation. As Dr. Tetlock states in his book, Superforecasting: The Art and Science of Prediction, on average, teams are 23% more accurate than individuals. That’s why our Superforecasters work as teams, bringing out the best in each other. However, to celebrate the New Year, we broke this rule for this piece, presenting individual forecasts as opposed to well-reasoned aggregates. To see our forecasting work, please visit Good Judgment’s Public Dashboard.