by Nick Rohrbaugh and Warren Hatch
The United Kingdom voted to Leave the European Union, sending shockwaves through world political capitals and financial centers. Most everyone expected a close vote, but few anticipated that Britain would vote to Leave.
Did the political and economic elites miscalculate the likelihood of a Leave vote? If so, they had good company. Most betting markets, as well as Good Judgment’s Superforecasters and participants on the GJ Open public forecasting site, closed with odds that favored a victory for the Remain camp.
Unless a forecaster assigns a 0% chance to an outcome that does in fact occur or 100% to an event that never happens, it’s impossible to judge the accuracy of a single forecast as being definitively “wrong.” How, then, can we evaluate whether the elites and other forecasters were wrong or just unlucky?
In other words, was the Leave vote simply a low-probability outcome that nonetheless occurred? Or did most people underestimate the “true” probability of a Leave vote? An after-the-fact “post-mortem” analysis of the Superforecasters’ predictions may shed light on this question. Top forecasters regularly undertake post-mortems, even when their predictions prove to be on the “right side of maybe,” to test the quality of their information search and reasoning. A thorough post-mortem on the Brexit forecasts will take time but can help identify lessons that apply more broadly and in so doing improve our collective understanding of other geopolitical risks.
Here’s some background. On the eve of the referendum, polls of voter intentions showed a tight race with Leave leading Remain 45% to 44%, and 11% of voters undecided. Various prediction markets displayed more confidence that the UK would vote to stay in the EU, estimating only a 24% chance of a Leave victory, roughly in line with 25% odds among online traders as well as the median of 25% of the 1,700 forecasters on the Good Judgment Open. For their part, Good Judgment’s panel of professional Superforecasters placed a probability of 24% for Britain’s voting to Leave the EU, or odds of roughly one in four.
When a good forecaster or good forecasting system predicts a 24% probability that an event will occur, we should expect that, on average, forecasts assigned a 24% probability will occur 24% of the time. A 24% forecast does not mean that there is a 0% chance that the event will occur, a distinction that all too often gets lost when forecasts are communicated in public. A properly calibrated forecaster knows that low-probability events will indeed occur some of the time, just as even the greatest batter expects to strike out occasionally. Knowing that a batter struck out once tells us almost nothing about that person’s batting average.
Keeping score over large numbers of forecasts helps forecasters correct both over- and under-confidence. For a single forecast, we must look to the underlying analysis to judge the quality of that forecast and to find ways to improve subsequent forecasts.
The book Superforecasting lists “Ten Commandments” that improve forecasting. Drawing from that list, here are some of the questions the Superforecasters are asking themselves to test the quality of the reasoning behind their predictions about the recent UK referendum. Please feel free to nominate other possible post-mortem questions HERE.
Properly Balancing Inside and Outside Views: Good forecasters often take what Daniel Kahneman and Amos Tversky call the outside view. Rather than focusing on the details that are specific to the case at hand, good forecasters identify a reference class of similar situations from the past, and anchor their initial forecast on the base rate, or the percentage of those cases where the event occurred. In previous independence referendums, such as when Scotland elected to remain as part of the UK or when Quebec voted not to declare sovereignty, the status quo has usually won the day.
Historical cases aren’t perfect predictors of future cases, though. Our post-mortem will consider whether forecasters over-relied on the outside view. In other words, did the unique political climate of this referendum merit giving more weight to case-specific or “inside-view” adjustments to the base rate derived from previous outcomes than the adjustments that our forecasters implicitly made in their predictions?
Under-reacting to New Evidence: Another way of thinking about the same point is to ask whether our forecasters–and others who made public predictions–were too slow to adjust their predictions in response to emerging information that might have signaled departures from the historical norm for independence referendums. For example, opinion polls suggested that those supporting Remain were typically younger than those supporting Leave. Younger voters historically have lower turnout rates than older voters, a pattern that seems to have prevailed in the UK referendum.
Did forecasters give enough consideration to demographics and turnout patterns when updating their predictions? Were there other case-specific indicators (such as the results of certain types of polls) that would have justified moving further away from forecasts anchored on historical base rates? The Superforecasters’ post-mortem will be looking at the weight given to such information when updating their forecasts.
Harnessing the Best Qualities of Crowdsourcing: It’s easy for forecasters to be swayed by repeated and vivid storylines such as media coverage suggesting that the Jo Cox killing increased the odds the Remain campaign would prevail. The Good Judgment Project’s research suggests that forecasting in small teams, where forecasters challenge one another’s positions while collaborating to make the most accurate forecasts possible, can produce better results than those achieved by forecasters working in isolation. Our Superforecasters take this approach, but we’ll be taking a close look at our internal debates to make sure that those who saw reasons for assigning higher probabilities to a Leave vote spoke up and that their colleagues gave their points due consideration.
The outcome of the referendum was a surprise to many, including many of the Superforecasters. But forecasting accuracy can only be determined over many questions. You can follow @GJ_Analytics on Twitter for daily forecasts from the Superforecasters on other events. We encourage you to keep tracking our accuracy and comparing your own forecasts with markets, pundits, and the Superforecasters. And if you think you can do better, join the thousands of other forecasters on Good Judgment Open to put your predictions to the test.