You might call 2016 the “Year of the Forecast” in US politics, as a seemingly unprecedented number of sources made public probability forecasts about the presidential election. Traditional media organizations, “data journalism” sites, academics, and crowd forecasting sites (including our own Good Judgment Open) went toe-to-toe to see who could most accurately predict the outcome of the election. Now that the results are in, it’s time to keep score – who was the most accurate?
At the Washington Post’s Monkey Cage blog, Pavel Atanasov and Regina Joseph tally the results from five sites that shared their daily forecasts and calculate a Brier score (a measure of the accuracy of a probabilistic forecast) for each.
They find that the Good Judgment Open crowd consensus was the most accurate forecast of the presidential election.
The GJ Open crowd gave Donald Trump a 24% chance of winning the election on November 8th, and even higher chances throughout much of the campaign. The combination of the crowd forecast’s conservatism and stability meant that Trump’s upset over Clinton put GJ Open ahead of Hypermind, PredictWise, DailyKos, and the Huffington Post – as well as several other sites that did not share data with the authors, such as the New York Times’ The Upshot, PredictIt, and the Princeton Election Consortium. The crowd forecast from GJ Open also proved to be much more stable than similarly conservative forecasts from FiveThirtyEight.
GJ Open also led or tied other sources in forecasting the outcome of several key swing states, including Florida, North Carolina, Ohio, Pennsylvania, Michigan, and Virginia.
The analysis shows crowd-based forecasting from sites like GJ Open and Hypermind outperformed statistical models based on polls (DailyKos and Huffington Post) or a combination of polls and prediction markets (PredictWise).
As the authors note, we can’t be sure that crowd-based forecasting methods will continue to outperform polls in future elections based on such a small sample. The accuracy of a forecast on one, or even a handful of questions is simply not reliable enough to draw strong conclusions about the forecaster’s long-term accuracy.
And, of course, all sources in the analysis predicted that Clinton was more likely to win the presidency than Trump and therefore scored worse than the proverbial “dart-throwing chimp” who would have forecasted a probability of 50% for each candidate. But in an era where statistical models abound, this election shows that we shouldn’t dismiss the wisdom of the crowd.