LAUNCHING THE “STATE OF THE UNION” CHALLENGE
A lot is happening on Good Judgment Open this week. We are wrapping the “First 100 Days” Challenge and launching the “State of the Union” forecasting challenge sponsored by the Washington Post’s Monkey Cage blog. “State of the Union” will offer a battery of questions on the major events and trends shaping U.S. politics through the 2018 midterm elections.
This is a great time to look back on what you can learn from your results in the “First 100 Days” challenge. Did the forecasting questions resolve as you expected? It’s time for a post-mortem on how you did.
To help Good Judgment Open forecasters craft their post-mortems, we asked the Superforecasters for their advice on how to conduct a post-mortem.
What have you learned from your successes and failures forecasting the first 100 days? Join the conversation on our Facebook page to learn from other forecasters’ post-mortems.
WHAT IS A “POST-MORTEM”?
A forecasting “post-mortem” is an analysis of the forecasting process for a question after this question has resolved. Philip Tetlock, one of the co-founders of Good Judgment Inc, recommended post-mortems as good forecasting practice in Superforecasting: The Art and Science of Prediction:
Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases. Don’t try to justify your failures. Own them! Conduct unflinching post-mortems: Where exactly did I go wrong? And remember that although the more common error is to learn too little from failure and to overlook flaws in your basic assumptions, it is also possible to [overcompensate]. Also don’t forget to do post-mortems on your successes too. Not all successes imply that your reasoning was right. You may have just lucked out by making offsetting errors. And if you keep confidently reasoning along the same lines, you are setting yourself up for a nasty surprise
Before we get into the Superforecasters’ thoughts on post-mortems, a note on the complexity of being “right” or “wrong” in forecasting. You’re only really, squarely “right” or “wrong” in forecasting if you make a prediction of 100% or 0% for an event and then the opposite occurs. You can learn more about this principle in Phil Tetlock’s talk on “Forecasting Failure” at Davos.
When it comes to post-mortems, Superforecasters tend to focus on errors in reasoning, such as those caused by cognitive biases. With this in mind, below you can find what some of our Superforecasters had to say on how they conduct unflinching post-mortems.
ADVICE FROM THE SUPERFORECASTERS
Learn from your errors in reasoning, even when you are “right”: “
“…To me, a post-mortem should be basically independent of my relative Brier score on that question. Instead, it should be focused on logical flaws in reasoning, gaps in information, or failure to understand the opposing argument.”
“… getting something right for the wrong reasons could be interesting. For example, let’s say on the Article 50 timing you forecasted it would happen in the right time frame because you thought the UK Supreme Court would come out the other way. Being right for the wrong reasons is interesting because your score won’t reflect an error and you might not internalize as you tend not to focus on questions you got right, as mentioned above”.
“I try to pay equal attention to questions I got right or wrong, simply trying to come back and see if the logic was sound and all facts were taken into account with a “proper” weight. Of course, the bias is always to dwell on what went wrong”.
Focus on errors in reasoning, ideally without knowing your score:
“I believe that hindsight bias is incredibly difficult to overcome, especially in post-mortems. To help combat hindsight bias, I try to write my personal post-mortem before I know my score relative to others. Once I know whether I did better or worse than others, I no longer trust my ability to assess my performance. In my mind, Brier scores on individual questions are useless for feedback – only assessments of my analysis before I know my score hold weight. Even then, I find it incredibly difficult to identify mistakes for most questions unless I made a clear logical error, had bad information, or failed to understand another forecaster’s argument.
If I had sound logic, good information, was aware of the opposing arguments, and properly identified the risks, then I generally feel good about my forecast – regardless of my Brier score.”
Look for lessons across several questions:
“… if it becomes clear what went wrong, which usually takes a couple of bad forecasts, that can be very helpful. A couple I can think of during the GJP tournament had to do with overcoming personal bias (and second guessing) when comes to leaders with an authoritarian bent. There was the Monti question where Berlusconi brought down the government although that was clearly not in Italy’s interests. The other question was about Shinzō Abe and whether he would visit the Yasukuni shrine. Such a visit would be bad PR for his country and especially at a time when there were negotiations coming up with China and South Korea. Yet, both these events did come to pass. Instead of paying enough attention to what these leaders had telegraphed in advance that they intended to do, I let my own assumptions about politicians adhering to norms, bias my judgment.”
Consider what you can learn from the bigger picture, not just your question score:
“For me, a good post-mortem comes from questions that haunt you, are ongoing and won’t close, and trouble you in your sleep. Examples for me are questions related to climate change, Israel-Palestine, and nuclear North Korea, that address matters beyond what the specific question framed and are matters of concern. Not so much post-mortem as in limbo or uncanny, unheimlich…”
Post-mortems are, in the end, about learning:
The best post-mortem is the one from which you learn the most. New facts, new approaches to estimates, new ways to cope with inherent bias, new ways to combine team resources, etc.
Armed with this information from the Superforecasters, navigate over to our Facebook page to reply to the latest post and share your post-mortems.