Nate Silver and the Risks of the “Wrong Side of Maybe”

Posted on Posted in Pundit Challenge

by Warren Hatch

It can be risky to put your neck out with precise probability estimates. Just ask Nate Silver.

Last September on CNN, he set these odds that Donald Trump or Ben Carson would end up getting the Republican nomination: “Maybe about 5 percent each, somewhere around there.”

This week, Slate took Nate Silver to task with this headline: “How Nate Silver Missed Donald Trump. The election guru said Trump had no shot.”

While 5% is a low probability, it’s more than “no shot.” This isn’t like George Tenet, the former CIA director, who said in 2012 that Iraq’s possession of weapons of mass destruction was a “slam dunk.” And even if it were, Slate should be giving credit for Mr Silver’s forecast on Ben Carson, whose presidential prospects have plunged sharply.

Mr Silver has periodically updated his Trump forecast, affirming low odds but remaining above zero. For instance, in November he ran through assorted scenarios on his website: “For my money, that adds up to Trump’s chances being higher than 0 but (considerably) less than 20 percent.” Slate cites these source but, alas, glosses over the nuance.

Now, Nate Silver could have said that Donald Trump will “maybe” be the nominee, as other pundits might have done, in which case he would have cover regardless of what happens: “maybe” straddles both sides of 50%. Instead Mr Silver gave a numerical probability estimate, putting him at risk of being on the wrong side of maybe in future headlines if Trump prevails.

In the book Superforecasting, Phil Tetlock and Dan Gardner document other cases of the  “wrong-side-of-maybe” fallacy. Back in 2012, for example, the prediction markets had 75% odds that the Supreme Court would strike down President Obama’s health care law. When the Court affirmed the law, New York Times reporter David Leonhardt declared that the wisdom of the crowds was flat out “wrong.”

For anything other than 0% and 100%, probabilities aren’t binary and their accuracy is something that is measured over time – whether it’s Nate Silver, the prediction markets, or the Superforecasters. Consider 100 different forecasts with a 5% probability. If those are accurate probability estimates, then we should expect 5 of them to resolve as “yes” and the other 95 as “no.” The key is to keep score so we can find out.

How much confidence should we lose if Nate Silver is too far on the wrong side of maybe? For the sake of argument, let’s assume that Mr Silver has a calibration score of 90% statistical confidence based on 500 forecasts. Let’s also run with his 5% forecast that Donald Trump will be the nominee. Finally, let’s assume that Donald Trump secures the Republican nomination (which is far from a foregone conclusion). What’s the best way for a Bayesian to adjust Nate Silver’s score? Submissions welcome!

Share on FacebookTweet about this on TwitterShare on LinkedInEmail this to someone

2 thoughts on “Nate Silver and the Risks of the “Wrong Side of Maybe”

  1. I actually am surprised in the opposite direction. I think that on the whole, Nate Silver has been underconfident in his models; he called many more predictions accurately than his stated confidence over time. This is true especially in the political arena, where his models have been very skillful very early in the election season, when he was already weighting uncertainty heavily. Given that, this single data point on the other side implies a less miscalibrated predictor than I would otherwise have assumed him to be. I will need to weight his numerical odds very slightly more heavily in the future – and it was a pretty high weight already.

  2. In my mind, this is a clear case of personal bias clouding one’s judgment. Happens all the time even to the best forecasters, who are human after all. Citing 5% estimate in September makes the unfortunate incident look better than it really was. 2% in August and “higher than 0 but (considerably) less than 20 percent” in late November are worse examples. (Note that unwillingness to come up with a number; the verbiage still implies single digit IMO).

Leave a Reply

Your email address will not be published. Required fields are marked *