On October 2nd, Colombian voters took to the polls to determine the fate of their government’s historic peace agreement meant to end 52 years of war with FARC rebels. The accord appeared likely to win approval, as polls showed that Colombians would vote “yes” in the plebiscite by a margin of almost two to one.
But when the votes were tallied, the verdict was clear – Colombians had narrowly rejected the plan for peace. With 37% of Colombians turning out to vote, 49.78% voted in favor of the measure while 50.21%, led by former president Álvaro Uribe, voted against it.
Why did the crowd (and the world at large) fail to foresee the outcome of the plebiscite? After a year of forecasting surprises – which saw many forecasters confidently failing to predict the nomination of Donald Trump and the UK’s Brexit vote – why didn’t we see more humility among forecasters about the prospect of peace in Colombia?
Some blamed the weather, as Hurricane Matthew battered the region and was likely a significant factor of the unexpectedly low voter turnout, especially in coastal regions where “yes” vote was already strong. Others noted a rural-urban divide, where more remote regions ravaged by the war tended to vote in favor of the deal, while more cosmopolitan areas voted against it. But our forecasters dug deeper in their post-mortem reflections, drawing out lessons that can be applied more broadly.
While most Colombians appeared to favor peace, many opposed the specific terms of the deal, feeling they were far too favorable to the FARC rebels – a sentiment echoed by Hugh Blanchard (Redcell2 on GJ Open), the most accurate forecaster on our question about the plebiscite:
Most of the people of Colombia did not approve the terms of the agreement. They felt (and continue to feel) that the FARC rebels were being treated far too generously in the terms of the proposed settlement. Especially offensive was the proposal that each and every FARC rebel would get $5000 or more from the government to stop fighting. Many people in Colombia have left lost family and friends in this protracted war. They don’t think the FARC should be rewarded for killing people, especially when the law-abiding citizens of Colombia will have to pay increased taxes to fund these payments.
Other forecasters agreed, including Samantha Lal (shiva), ranking fifth in accuracy score after forecasting a 1% chance of approval on the day of the vote:
I imagine myself, as a Colombian mother, girlfriend, brother, … as well as a FARC member. If I had lost someone to a violent outcome, knew other families who had too, and we still all had major questions unanswered like: “Why did they pick my son, father, neighbour, etc”, “Who did this (like a name and a face)”, and “How can I bury someone I still can’t find?” Closure after trauma is an integral part of human and animal nature and, well, I would be very reluctant, even suspicious, watching my government leader shake hands with the same people who were involved in this. … At least two generations had experienced La Violencia, and as a living memory which is much more vital – think Holocaust survivors being alive to tell their tale.So no because of a lack of closure, lack of justice. … Most shocking of all to me was that the discussions Santos had did not seek out the will of the people, or the hundreds of thousands of victims’ families. The government assumed their will was people’s will – not, as was their sworn duty, the other way around.
In micro economics most of the predictions are wrong because the market participants are assumed to be rational and logical. It is so difficult to go against the consensus when all the ‘experts on the ground write’ “Colombians expected to say ‘yes’ to peace”. In the case of Brexit I learned that people in the capital lived in the different world from the rest of the UK.
Many forecasters described “herding” of the media predicting a “yes” victory based on the same information, which was reflected in the GJO consensus. Samantha explained:
I think [GJ Open forecasters] follow the international web-based media, which in this case was wrong. For this question I did a lot of ‘reading between the lines’, to kind of get into the heads of the people who would be voting. That’s not quantifiable. There is heavy reliance on polls, stats, figures to lend weight to assumptions, especially in Europe and North America. It like giving a random thing a truth by numbers: “nine out of ten dentists recommend Colgate” Some of the quantitative aspects of stats and math are also just guesswork themselves.
Eric Ehrmann (Firedog), the second most accurate forecaster on the plebiscite question, agreed that forecasters and the media were too bold in assuming that the agreement was a done deal:
Generally speaking, I think a lot of those who participated in forecasting this problem concluded that the “For” vote was a “slam dunk.” With UN Secretary General Ban and U.S. President Obama giving the deal “two thumbs up” and President Santos being front loaded for the Nobel Peace Prize and a lot of upbeat chatter from mainstream and online media in the US, UK and Latin America, the referendum “For” seemed to be just a “rubber stamp.”
Tetlock and Gardner say that superforecasters have the ability to synthesize material from sources with very different outlooks on the world. That trait was not in evidence very much on this forecasting problem.
Dima Klenchin (Dima-K), a Good Judgment Inc Superforecaster who was also forecasting this question on GJ Open, described the difficulty of taking the emotional temperature of a nation into account when forecasting elections and referenda:
The parsimonious explanation to me is a version of Bradley effect. Who is against the peace? – Of course, nobody! But then, when it comes to polls and being along with the ballot, perhaps the vengeance and the feeling that murderers get off too easy takes over in large enough % of cases? … In an ideal world, one just finds out “how people feel” and tries to correct the polls (or newspapers’ lies, as would the case in Sierra Leone). But there is no known recipe for that. Pollsters themselves would kill for this ability! It’s just *that* hard. The now classic example that I always give: The fact that Nigel Farage so easily proclaimed a defeat within the first two hours of Brexit vote counting shows that he himself had no good idea how people in his own country truly feel about EU and immigration.
So what can forecasters learn from this miss? How can we avoid overconfidence in our forecasts? Hugh suggests relying more on local sources:
Read the local papers and talk to the people, not just government pronouncements in “official” position papers and government-leaning media. … Talk with and listen to the people, the often ignored “man in the street.”
Samantha encourages looking at the polls as people, not just numbers, in order to consider how and when they might be wrong:
Referendums, plebiscites and even polling increments all boil down to just many, many single points of data which reveal themselves to be the thoughts and actions of one person multiplied many times. Instead of looking at the big picture – the aggregate of all the data points (or little specks of people), when it comes to refs, plebs, polls, these are random individuals who all share the same opinion on particular issue. … When you start to think of it like that, as points of data that equal individuals, things like feelings become central to the question. I couldn’t imagine a mother whose son was taken by FARC to be a child soldier getting over that with no trial, no consequences and, worse, seats (or power) in parliament for the takers. Also thinking of it this way, means random things like the weather, or when the school term breaks as per Brexit, become viable reasons why a vote might go in a different direction.
Dima called for humility when forecasting unfamiliar topics:
If you don’t know the people and the country well, be humble and don’t over-commit. … I see people making forecasts about Russia going with the obviously bone-headed anti-Russian crap propaganda and I can only shake my head looking at it. But then I proceed to make a confident forecast about presidential election in Nigeria based on the idea that as far as elections in Africa are concerned, one factor overrides all others. Stupid!
Likewise, in Colombia: even without the firm polls, the bias was incredibly heavily toward “Yes”. The idea behind it, best I can tell, goes like that: Of course there is a fix because if there were no fix, Santos would not go ahead with it.
But: the reality is that most countries are about as “democratic” as we are. That is, the myth of “democracy” aside, the elites in those countries do not control masses any more than they do in good ole USA.
So, the bottom line: unless you know the place and its peculiarities very well, always double or triple the uncertainty in the standard signals like polls and economy/incumbency factor/”common sense.” (Once again, easier said than done.)
While this version of the peace agreement will not be implemented, Colombians have not given up on ending the war and securing peace. Both President Santos and Rodrigo Londoño, the leader of FARC, have reiterated their desire for peace and that the ceasefire that began in August would continue.
Do you have thoughts on why so many failed to predict the outcome of Colombia’s plebiscite? Add your comments below and on our Facebook page, and be sure to join Good Judgment Open and check out our questions on elections and referenda around the world.