Handicapping the odds

Handicapping the odds: What gamblers can learn from Superforecasters

Successful gamblers, like good forecasters, need to be able to translate hunches into numeric probabilities. For most people, however, this skill is not innate. It requires cultivation and practice.

In Superforecasting: The Art and Science of Prediction, a best-selling book co-authored with Dan Gardner, Phil Tetlock writes: “Nuance matters. The more degrees of uncertainty you can distinguish, the better a forecaster you are likely to be. As in poker, you have an advantage if you are better than your competitors at separating 60/40 bets from 40/60—or 55/45 from 45/55.”

Good Judgment’s professional Superforecasters excel at this, but thinking in probabilities doesn’t come naturally to the majority of human beings. Daniel Kahneman and Amos Tversky, who studied decision making under risk, found that most people tend to overweight low probabilities (e.g., the odds of winning a lottery) and underweight other outcomes that were probable but not certain. In other words, people on average evaluate probabilities incorrectly even when making critical decisions.

Superforecasting gambling poker
Base rate neglect often leads to poor decisions in forecasting, finance, or gambling.

If you’ve participated in any Good Judgment training, you’ll know that the first step in estimating correct probabilities is to identify the base rate—the underlying likelihood of an event. This is also the step that the majority of decision makers tend to ignore. Base rate neglect is one of the most common cognitive biases we see in training programs and workshops, and it generally leads to poor investing, betting, and forecasting outcomes.

For those new to the concept, consider this classic example: “Steve is very shy and withdrawn, invariably helpful, but with little interest in people or the social world. A meek and tidy soul, he has a need for order and structure and a passion for detail.”

Is Steve more likely to be a librarian or a farmer? A librarian or a salesman?

While the description, offered in Daniel Kahneman’s Thinking, Fast and Slow, may be that of a stereotypical librarian, Steve is in fact 20 times more likely to be a farmer—and 83 times more likely to be a salesman—than a librarian. There are simply a lot more farmers and sales persons in the United States than male librarians.

Base rate neglect is the mind’s irrational tendency to disregard the underlying odds. Failure to account for the base rate could lead, for example, to the belief that participating in commercial forms of gambling is a good way of making money. Likewise, failure to factor in the house edge could lead to poor betting decisions.

Fortunately, the mind’s tendency to overlook the base rate can be corrected with training and practice.

Recognition of bias and noise, and techniques to mitigate their detrimental effects, should be at the heart of any training on better decision making. In Good Judgment workshops, we have consistently observed tangible improvements in the quality of forecasting as a result of debiasing interventions.

The other essential component is practice. On Good Judgment Inc’s public platform, GJ Open, anyone can try their hand at forecasting—from predicting the next NBA winner to estimating the future price of a bitcoin. Unsurprisingly, those forecasters who use base rates and forecast on the platform regularly tend to have better results.

To stay on top, gamblers, like successful forecasters and professional Superforecasters, need to actively seek out the base rate and mitigate other cognitive biases that interfere with their judgment. While “Thinking in Bets,” as professional gambler Annie Duke puts it in her best-seller, does not come easy to most people, better decision making—in forecasting, investing, and gambling alike—is a skill that can be learned. With an awareness of cognitive biases, debiasing techniques, and regular practice, anyone can acquire the mental discipline to handicap the odds more effectively.

* This article originally appeared in Luckbox Magazine and is shared with their permission.

Forecasting the Tokyo Olympics

Forecasting the Tokyo Olympics

In late July 2020, a year ahead of the Tokyo Olympics (postponed in 2020 and scheduled to open 23 July 2021), we asked the Superforecasters whether the Games will begin as planned. By 7 September 2020, the Superforecasters had a clear answer. Back then, they gave the Games a 63% probability of proceeding and have hardly looked back.

The picture was by no means clear if you followed media reports around the Olympics throughout the past year. The IOC and the organizing committee were adamant that options such as a cancellation or delay were off the table. The Japanese public became increasingly opposed to the event. COVID-19 hit the country with a new wave. A range of dissonant headlines, speculations, public opinion polls, and even allegations that Japan had privately decided to cancel the Games (Times) all contributed to the noise surrounding the future of the event.

Good Judgment’s professional Superforecasters are skilled at separating the signal from the noise. They took into account such factors as the associated costs; the likelihood that a vaccine would be developed, tested, and become available by the time of the event (this was months before any COVID-19 vaccine was found to be effective—and safe—in a large clinical trial); and the increasing international experience with measures to contain risk. See how their forecast evolved over time against the backdrop of media reports throughout the year.

A list of media headlines and key events is provided at the bottom of this article, demonstrating both the signal and the noise surrounding the Tokyo Olympics.

For more examples of our work, you can find a sample of current questions posed to the Superforecasters and see their forecasts on our Public Dashboard. Full access to the Superforecasts and commentary is available through subscription via our FutureFirst™ monitor.

Tokyo Olympics: A Sample of Media Headlines Since July 2020

21 July 2020: USA Today: “As COVID-19 pandemic rages on, experts say it’s unlikely Tokyo Olympics can be held next summer”

20 Aug 2020: Japan Times: “Majority of Japanese firms against holding Tokyo Olympics in 2021”

7 Sept 2020: BBC: “Games will go ahead ‘with or without Covid’, says IOC VP”

1 Oct 2020: The Diplomat: “The International Olympic Committee has ruled out postponing the Tokyo Games for a second time”

1 Dec 2020: “Report: Delay of 2020 Tokyo Olympics cost $3 billion”

15 Dec 2020: Japan Times: “Most in Japan oppose holding Olympics in 2021, polls show”

27 Dec 2020: Kyodo: “Pandemic causing uncertainty, unease for Tokyo Olympic ‘host towns’”

7 Jan 2021: BBC: “Tokyo 2020: No guarantee Olympics will go ahead, says IOC’s Pound”

10 Jan 2021: Kyodo: “About 80% favor canceling, postponing Tokyo Olympics in summer: poll”

11 Jan 2021: Japan declares a state of emergency

13 Jan 2021: Guardian: “Tokyo’s Covid outbreak adds to doubts over hosting Olympic Games”

15 Jan 2021: NYT: “Hopes for Tokyo’s Summer Olympics Darken”

CBS: “Tokyo Olympics 2021: Spike in COVID-19 cases has Japanese officials bracing for possible postponement”

19 Jan 2021: AP: “Tokyo Olympics Q&A: 6 months out and murmurs of cancellation”

BBC: “Tokyo Olympics ‘unlikely to go ahead in 2021’”

21 Jan 2021: Times: “The Japanese government has privately concluded that the Tokyo Olympics will have to be cancelled because of the coronavirus”

22 Jan 2021: Reuters: “Japan and IOC deny that Olympics will be cancelled”

11 Feb 2021: Guardian: “Tokyo 2020 Olympics president expected to resign over sexist comments”

15 Feb 2021: CNN: “An earthquake at the Olympic torch relay start point is just the beleaguered Tokyo 2020 Games’ latest crisis”

17 Feb 2021: Seiko Hashimoto becomes new president of the Tokyo Olympic organizing committee

9 March 2021: Kyodo: “Japan to stage Tokyo Olympics without overseas spectators”

25 March 2021: Tokyo Olympic torch relay begins

15 April 2021: Washington Post: “Olympics could be canceled because of virus, Japan ruling party figure admits”

1 May 2021: Washington Post: “Olympic officials are determined to have a Tokyo Games despite Japan’s growing doubts”

12 May 2021: BBC: “Tokyo 2020: United States track and field team cancels pre-Olympic training in Japan”

14 May 2021: NPR: “Opposition to Tokyo Games Grows Heated amid COVID Concerns”

Guardian: “Hospitals overwhelmed as Covid cases surge in Osaka”

18 May 2021: CNBC: “Tokyo medical association calls for cancellation of Tokyo Olympics due to spike in Covid cases”

19 May 2021: CNN: “Dozens of Japanese towns have canceled plans to host foreign athletes from around the world due to concern over Covid-19”

25 May 2021: CNN: “Canceling Tokyo Olympics is ‘essentially off the table,’ says IOC member Dick Pound”

2 June 2021: AP: “Yes. Tokyo Olympics are ‘a go’ despite opposition, pandemic”

13 June 2021: CBS: “Cancel the Tokyo Olympics? It’s unlikely. Here’s why”

Question Clustering

Question Clustering: Ensuring relevance and rigor in business and geopolitical forecasting

Forecasting is an essential part of business. Companies use historical data and economic trends to make informed estimates of future sales, profits, and losses. Amazon and Google rely on predictive algorithms to highlight specific products or search results for their customers. Shopkeepers arrange their display windows based on their predictions of demand. Some of the forecasting is narrow and company-specific. The more challenging part has to do with broader questions, such as the overall market outlook.

Consider this question: What will market conditions be like after the pandemic?

This question is of great relevance to any business decision-maker, but it’s so broad that it could be approached from many different angles, using multiple definitions. “Market conditions,” for instance, involves multiple factors, from industry-specific trends to interest rates, from consumer spending to supply chains. How do we intend to measure this? What time frame are we looking at?

Combining related forecasting questions into clusters can reveal bigger-picture insights for businesses and industries

Much more tractable is a narrower question—”On 31 January 2022, what will be the average US tariff rate on imports from China?” But in checking all the boxes to craft a rigorous question, narrow topics may lose their relevance to decision-makers trying to determine the prospects for their business down the line. This is the rigor-relevance trade-off.

To tackle the relevant strategic questions that decision-makers ask with the rigor that accurate forecasting requires, Good Judgment crafts sets of questions about discrete events that, in combination, shed light on a broader topic. Good Judgment calls this technique question clusters. Good clusters will examine the strategic question from multiple perspectives—political, economic, financial, security, and informational. These days, a public health perspective is also usually worthwhile. Aggregating the probabilities that the Superforecasters assign to such specific questions generates a comprehensive forecast about the strategic question, as well as early warnings about the deeper geopolitical or business trends underway.

The Good Judgment team developed this method in a research study sponsored and validated by the US Intelligence Advanced Research Projects Activity from 2011-2015, where the Superforecasters outperformed the collective forecasts of intelligence analysts in the US government by 30%. Since then, the Superforecasters have refined and expanded their approach to evaluate emerging consumer trends, inform product development, and understand the factors driving commodity markets. A similar framework was detailed in a recent Foreign Affairs cover article by Professor Philip Tetlock and Dr Peter Scoblic.

As an example, a question cluster on a critical topic that clients have asked Good Judgment to develop includes political, military, social, and economic forecasts on emerging trends regarding Taiwan:

    • Will Taiwan accuse the People’s Republic of China (PRC) of flying a military aircraft over the territory of the main island of Taiwan without permission before 31 December 2021?
    • Will the PRC or Taiwan accuse the others’ military or civilian forces of firing upon its own military or civilian forces before 1 January 2022?
    • Before 1 January 2022, will US legislation explicitly authorizing the president to use the armed forces to defend Taiwan from a military attack from the PRC become law?
    • Will Taiwan (Chinese Taipei) send any athletes to compete in the 2022 Winter Olympics in Beijing?
    • Will the Council of the EU adopt a decision authorizing the Commission to open negotiations with Taiwan on an investment agreement?
    • Will the World Health Organization reinstate Taiwan’s observer status before 1 January 2022?

As another example, in forecasting the technological landscape, the Superforecasters estimate the emerging trends in electric car sales, hydrogen-fueled vehicles, the growth of Starlink services, and social media regulation, among others.

Independently, these forecasts are crucial for some companies and investors and informative for the discerning public. Taken together, they point to trends that are shaping the future of the geopolitical and business world.

The same technique can be applied to Amazon. The traditional approach of security analysts is to build a fixed model that predicts valuation based on dividends, cash flow, and other objective metrics. A question cluster approach can uncover critical subjective variables and quantify them, such as the risk of a regulatory crackdown or shifts in labor relations, to make the analyst’s model more robust.

“In business, good forecasting can be the difference between prosperity and bankruptcy,” says co-founder of Good Judgment, Professor Philip Tetlock. Successful businesses rely on forecasting to make better decisions. Using clusters of interrelated questions is one way to ensure those forecasts are both rigorous and relevant.

* This article originally appeared in Luckbox Magazine and is shared with their permission.