Informed Practice and Superforecasting: Taking Your Forecasts to the Next Level

Written by

Informed Practice and Superforecasting: Taking Your Forecasts to the Next Level

“Not all practice improves skill. It needs to be informed practice.”
– Phil Tetlock & Dan Gardner in Superforecasting

In any area of decision-making where uncertainty looms large, accuracy is the gold standard. However, many decision makers often find themselves in a frustrating cycle—sometimes they make the right call, but other times they miss the mark entirely. Inconsistency can be costly. So, what separates those who occasionally succeed from those who reliably deliver top-notch forecasts? The answer lies in informed practice—one of the concepts at the heart of Superforecasting.

What Is Informed Practice?

Informed practice is not just repetition. It’s a deliberate and thoughtful process of learning from each forecast, refining techniques, and continuously updating one’s beliefs based on new information. It’s about approaching forecasting with a Superforecaster’s mindset—an outlook geared toward improvement, with a consistent effort to mitigate one’s cognitive biases.

What Can Forecasters Learn from Superforecasters?

Superforecasters, known for their uncanny forecasting accuracy, exemplify informed practice. They don’t pull numbers out of a hat or look into a crystal ball for answers. For every question they face, they engage in a rigorous process of analysis, reflection, and adjustment. Here’s how informed practice gives them the edge:

1. Learning from Feedback: Superforecasters thrive on feedback. They meticulously track their forecasts, comparing them against the outcomes to identify where they went right and where they missed the mark. This feedback loop is crucial. It allows them to recalibrate their approach and avoid making the same mistakes twice. Over time, this leads to more refined and accurate forecasts.

2. Understanding Probability: A key aspect of informed practice is the understanding and effective use of probability. Superforecasters don’t think in black-and-white, yes-or-no terms. They consider a range of possible outcomes and assign probabilities to each. They also update these probabilities as new information becomes available, a process known as Bayesian reasoning. This probabilistic thinking helps them navigate uncertainty with greater precision.

3. Continuous Learning: The world is constantly changing, and so too are the variables that influence forecasts. Superforecasters are voracious learners, continuously updating their knowledge base. They stay informed about the latest developments in multiple areas, thus grounding their forecasts in the most current data and insights.

4. Mitigating Cognitive Biases: Cognitive biases can cloud judgment and lead to poor forecasts. Superforecasters are keenly aware of these biases and actively work to mitigate their impact. Through informed practice, they develop strategies to counteract such biases as overconfidence, anchoring, confirmation bias, and more, to make well-calibrated forecasts.

What Is the Role of Collaboration in This?

Informed practice is not a solitary endeavor. Collaboration with other forecasters is a powerful tool for improving accuracy and keeping track. By engaging in discussions, comparing notes, and challenging each other’s assumptions, forecasters can gain new perspectives and insights. Good Judgment’s Superforecasters work in teams, leveraging the collective intelligence of the group to arrive at superior forecasts.

What Practical Steps Can I Take?

1. Keep Track: Keep a record of your forecasts and compare them with the outcomes. Analyze your hits and misses to identify patterns and areas for improvement.

2. Seek Feedback: Seek out feedback from peers or through forecasting platforms such as GJ Open, which provides performance metrics. Use this feedback to refine your approach.

3. Diversify Your Sources of Information: Regularly update your knowledge on the topics you forecast and seek out diverse sources. This includes staying current with news, research, and expert opinions, including those you disagree with.

4. Practice Probabilistic Thinking: Assign probabilities to your forecasts and be willing to adjust them as new information emerges. This helps you avoid the trap of binary thinking.

5. Challenge Your Assumptions: Regularly question your assumptions and be open to changing your mind. This flexibility is crucial in a rapidly changing world.

6. Get a Head Start with GJ Superforecasting Workshops: Consider enrolling in a Superforecasting workshop. Good Judgment’s workshops, led by Superforecasters and GJ data scientists, leverage our years of experience in the field of elite forecasting as well as new developments in the art and science of decision-making to provide you with structured guidance on improving your forecasting skills. Our practical exercises will boost your informed practice, offering you lifelong benefits.

Informed practice is the cornerstone of good forecasting and one of the secrets behind the success of Superforecasters. By diligently applying the above principles, you can enhance your forecasting skills and make better-informed decisions. See the workshops we offer to help you and your team take your forecasting success to the next level.

Superforecaster Tips: Dealing with Confirmation Bias in Election Forecasting

Written by

Superforecaster Tips: Dealing with Confirmation Bias in Election Forecasting

As the 2024 US election approaches, forecasters are faced with the daunting task of finding signal amid a cacophony of partisan noise, personal biases, and volatile public opinion. One significant challenge is confirmation bias—the tendency to search for, interpret, and recall information in a way that confirms one’s preconceptions. In this blog post, we draw on an internal discussion among seasoned Superforecasters to explore practical strategies forecasters can use to mitigate confirmation bias in election forecasting.

Diversifying Information Sources

“Assign yourself to spend some time reading (reasonably reputable) news sources that disagree with your general perspective on the question.”

Superforecasters highlight the importance of consuming a balanced diet of news sources, including those that challenge one’s beliefs. This approach was systematized by Good Judgment Project (GJP) superforecaster Doug Lorch, who wrote a program to randomize his news intake among a diverse set of sources.

“It certainly didn’t hurt,” recalls Terry Murray, CEO Emeritus of Good Judgment Inc and Project Manager for the GJP at UC–Berkeley. “He was the top forecaster in the whole IARPA tournament that year.”

Engaging in Scenario Analysis and Premortems

“I try to run through various scenarios where [the expected winner] could end up losing.”

Superforecasters routinely consider alternative outcomes by rigorously testing their own assumptions and logic. This involves running through various scenarios where expected outcomes might not materialize and thinking critically about the conditions that would lead to different results.

Embracing Epistemic Humility

“One thing I know is that I don’t know much.”

Acknowledging the limits of one’s knowledge and being open to new information is another tip the Superforecasters offer. This strategy is crucial for preventing overconfidence and being receptive to counterarguments.

Red Teaming

“One of the most important duties for me, as a Red Team member, is not to convince a forecaster that they are wrong… Rather, it’s to test the confidence of the Superforecaster in their own forecast.”

Having a red team to challenge forecasts helps forecasters to re-evaluate the confidence in their arguments and consider why they might be wrong. Red teaming is a standard practice in all Good Judgment’s forecasting.

Leveraging Collective Wisdom

“Sometimes, it pays to listen to the articulated reason of an outlier.”

Some Superforecasters use the median forecast of their group as a benchmark, particularly when their individual estimates deviate significantly from the consensus. This approach can provide a reality check against one’s own extremes. It is important, however, to pay attention to outlier opinions too, to resist conformity and groupthink.

As we dive into another election cycle, the discipline of forecasting reminds us that remaining actively open-minded is more crucial than ever. Combating confirmation bias in election forecasting is no small feat, given the complexity and the emotionally charged nature of politics. However, by employing strategies such as diversifying information sources, engaging in premortems, practicing epistemic humility, employing red teaming, and referencing the collective wisdom of peers, forecasters can enhance the accuracy and reliability of their predictions. Good Judgment’s exclusive forecast monitoring tool FutureFirst™ offers daily forecast updates on election results and trends and many other topics, brought to you by professional Superforecasters.

Learn More about FutureFirst™!

Decoding SCOTUS: Navigating Media Bias in Supreme Court Forecasting

Written by

Decoding SCOTUS: Navigating Media Bias in Supreme Court Forecasting

Superforecaster, GJ managing director, and leader of Good Judgment’s question team, Ryan Adler shares tips on how to approach forecasting Supreme Court decisions.

May is a lovely time of year. Lots of daylight, newborn critters all around, and Supreme Court junkies start to get their fix from now through the end of June. As someone who has watched the Court enthusiastically for more than a quarter of a century, I always look forward to decision season. However, in all those years, I’ve learned a painful lesson. The press is generally terrible at reporting about the Court. For Court junkies like me, it’s not that big of a deal. I learned early in my studies that if I want to know what a case means, I need to dig into the weeds myself. But as someone who has also written dozens of SCOTUS forecasting questions, I know that many forecasters don’t have the benefit of having read thousands of pages of appellate decisions to instruct their analysis. So, as is the case for anybody forecasting on a topic with which they aren’t intimately familiar, they lean on press reporting to fill in the gaps.

“Creating an emotional investment in the outcome of a case is poison for a forecaster.”

Alas, most journalists and talking heads intent to tell readers and viewers “how it is” end up just regurgitating heavily politicized schtick from one side of the aisle or the other. A glaring example is Trump v. United States, the presidential immunity case out of the DC Circuit Court of Appeals. For the vast majority of the press, it’s a question of what the Court will let happen to Trump in Jack Smith’s federal election interference case. For anyone who understands the Constitution and listened to oral arguments in the case, the Court knows it’s grappling with something much bigger and fundamental than Trump’s actions and statements on January 6th. Questions of presidential immunity go to the heart of the structure of the US’ separation of powers, perhaps the greatest innovation from the Founding Fathers that has proven key to good governance in modernity. Framing the case as democracy in peril (you can find such arguments from both Trump’s supporters and detractors) is, in this writer’s opinion, theater. This isn’t to say that the events of January 6th are nothing of concern, nor does it make light of the potential precedents to be set. But let’s be honest: if the party affiliation of the defendant in this case were flipped, who honestly thinks that the arguments from the “legal experts” in the media wouldn’t look far different? That doesn’t make these people liars, but it highlights that these “experts” are doing something very different than what we ask forecasters to do—figure out what will happen irrespective of what they feel should happen. Such “experts” placate and/or infuriate, which is how politics works.

What is to be done?

Creating an emotional investment in the outcome of a case is poison for a forecaster. We can debate all day about the political implications of a Court decision, and the nine robed ones don’t operate in a vacuum. That all notwithstanding, there is a stark difference between being mindful of political factors that influence the conversation among them and acting as though those implications represent the sole axis upon which the case will turn. It may seem hard not to see anything and everything through a political lens, and most media outlets have given up pretending that their reporting isn’t first run through a political prism.

So, what is to be done? As James Madison stated that ambition must be made to counteract ambition, this writer suggests that cynicism must be made to counteract cynicism. Take the words of any legal “expert” or talking head with a large grain of salt. Being a lawyer doesn’t make you an expert on the Constitution. Many lawyers who practice would probably admit that the last time they paid attention to constitutional law was prepping for a bar exam. That fact is not mitigated by gaining the title of “contributor.” If what you read or hear feels like it’s what you may personally want to read or hear, be suspicious. That’s sound advice for anything in the media, but it’s especially crucial for navigating the flood of gibberish that inundates the airwaves and web when a high-profile case makes it to Washington.

Do you have what it takes to be a Superforecaster? Find out on GJ Open!