by Warren Hatch
It can be risky to put your neck out with precise probability estimates. Just ask Nate Silver.
Last September on CNN, he set these odds that Donald Trump or Ben Carson would end up getting the Republican nomination: “Maybe about 5 percent each, somewhere around there.”
This week, Slate took Nate Silver to task with this headline: “How Nate Silver Missed Donald Trump. The election guru said Trump had no shot.”
While 5% is a low probability, it’s more than “no shot.” This isn’t like George Tenet, the former CIA director, who said in 2012 that Iraq’s possession of weapons of mass destruction was a “slam dunk.” And even if it were, Slate should be giving credit for Mr Silver’s forecast on Ben Carson, whose presidential prospects have plunged sharply.
Mr Silver has periodically updated his Trump forecast, affirming low odds but remaining above zero. For instance, in November he ran through assorted scenarios on his website: “For my money, that adds up to Trump’s chances being higher than 0 but (considerably) less than 20 percent.” Slate cites these source but, alas, glosses over the nuance.
Now, Nate Silver could have said that Donald Trump will “maybe” be the nominee, as other pundits might have done, in which case he would have cover regardless of what happens: “maybe” straddles both sides of 50%. Instead Mr Silver gave a numerical probability estimate, putting him at risk of being on the wrong side of maybe in future headlines if Trump prevails.
In the book Superforecasting, Phil Tetlock and Dan Gardner document other cases of the “wrong-side-of-maybe” fallacy. Back in 2012, for example, the prediction markets had 75% odds that the Supreme Court would strike down President Obama’s health care law. When the Court affirmed the law, New York Times reporter David Leonhardt declared that the wisdom of the crowds was flat out “wrong.”
For anything other than 0% and 100%, probabilities aren’t binary and their accuracy is something that is measured over time – whether it’s Nate Silver, the prediction markets, or the Superforecasters. Consider 100 different forecasts with a 5% probability. If those are accurate probability estimates, then we should expect 5 of them to resolve as “yes” and the other 95 as “no.” The key is to keep score so we can find out.
How much confidence should we lose if Nate Silver is too far on the wrong side of maybe? For the sake of argument, let’s assume that Mr Silver has a calibration score of 90% statistical confidence based on 500 forecasts. Let’s also run with his 5% forecast that Donald Trump will be the nominee. Finally, let’s assume that Donald Trump secures the Republican nomination (which is far from a foregone conclusion). What’s the best way for a Bayesian to adjust Nate Silver’s score? Submissions welcome!