The HFC Challenge has wrapped up on Good Judgment Open, and Good Judgment congratulates all HFC Challenge forecasters warmly, as we prepare to begin the official Hybrid Forecasting Competition in March 2018.
Featuring over 100 questions over 5 months, the HFC Challenge attracted a total of 1,869 forecasters. HFC Challenge participants forecasted important political and economic questions, to better prepare for the official Hybrid Forecasting Competition. HFC Challenge questions covered a diversity of topics included geopolitics, public health, military conflict, and economics.
The leaderboard stacked up as follows (listing each forecaster’s username):
Here’s a rundown of some important stats we kept an eye on throughout the Challenge:
What questions did the HFC Challenge forecasting crowd do “best” on? (i.e., Which had the highest mean accuracy scores?)
1- “Will ACLED record a riot/protest event in the Cote d’Ivoire (Ivory Coast) for August 2017?
2- “Will ACLED record any civilian fatalities in Tunisia between 15 October 2017 and 31 October 2017?”
What questions did the HFC Challenge forecasting crowd struggle the most with? (i.e., Which had the lowest mean accuracy scores?)
1- “What will be the daily closing spot price of Brent crude oil (USD per barrel) on 1 December 2017, according to the U.S. EIA?”
2- “What will be the short-term interest rate for Mexico (MEX) in December 2017?”
What question were users who read Superforecasting most accurate on?
“Will FEWS NET publish a Food Security Alert with ‘famine’ and ‘South Sudan’ or ‘Nigeria’ in its headline between 6 September 2017 and 25 October 2017?”
What question were users who had participated in the Good Judgment Project most accurate on?
“Will ACLED record a riot/protest event in the Cote d’Ivoire (Ivory Coast) for August 2017?
What was the most volatile question?
“What will be the short-term interest rate for Sweden (SWE) in December 2017?”
To capture individual reactions to the Challenge, Good Judgment reached out to the HFC Challenge’s top-scoring forecasters.
The top-scoring forecaster, dsarin, was interested in HFC’s overall goals: “The aim of the challenge was interesting. Humans lost in chess, poker, Go… Forecasting could be the last frontier (before that scene from Terminator 2)! I am very curious to see if the machines could enhance the forecasting accuracy.”
Meanwhile, the HFC Challenge’s second forecaster, cmeinel, was highly motivated to learn from forecaster teammates’ complimentary skills: “I have been amazed at how many various and complementary strategies my teammates have revealed to me.”
(Check out the full conversations with the #1 and #2 Challenge leader below.)
Good Judgment and IARPA would like to thank all 1,869 participating HFC Challenge forecasters, who contributed a remarkable number of forecasts –26,663!
The successor to the HFC Challenge – the official Hybrid Forecasting Competition – is scheduled to begin in March 2018. The Hybrid Forecasting Competition is an IARPA-sponsored research tournament that will test whether AI and advanced algorithms can help improve the accuracy of human forecasting.
You + AI = better forecasts? Let’s find out! Research volunteers are still needed for the official Hybrid Forecasting Competition. It’s not too late – learn more and register here.
Conversation with Dsarin, #1 Forecaster on the HFC Leaderboard
1. Why did you participate in the HFC Challenge on GJOpen.com?
The aim of the challenge was interesting. Humans lost in chess, poker, Go… Forecasting could be the last frontier (before that scene from Terminator 2)!
I am very curious to see if the machines could enhance the forecasting accuracy.
2. Do you have any thoughts on if you learned anything from the HFC Challenge?
The HFC questions were diverse and interesting (Thank you!). I was introduced to a variety [of] great databases from earthquakes to famines, to hacks and even atrocities (which was the worst part). The batch of 50 new questions uploaded at the same time was a bit overwhelming, however the diversity of subjects and learning something new helped with motivation.
The most disturbing question was on civilian deaths in Syria. Tallying deaths on daily basis and reading a bit of information about them provided by SNHR was disheartening. This should be a compulsory reading for all ‘policymakers’.
3. Were there any interesting questions that stood out – maybe a particularly challenging one from the HFC Challenge? If so, how did you tackle it?
Most of the questions were approachable with lots of data to calculate base rates. Interest rates, commodity prices, approval ratings and even earthquakes. There were plenty of data and sufficient materials to understand the relationships between different variables. The question that stood out was about the removal of Soviet memorials in Poland as it was about the new law coming into effect post the question’s start date, external pressures to do nothing and internal pressures to implement the law immediately. Google translate was vital for the question… I monitored Polish and Russian news articles, found municipalities most likely to implement the law, looked at the websites of different municipalities responsible for implementing the law, monitored government contracts for “monument removals”. I also nagged my Polish friends to call municipalities to find out the exact removal dates (after identifying a few likely targets). In one instance we got through to the architect responsible for the removal of the monument and redecoration of the square who assured us that the monument in question will be removed but the timing of the removal was halfway before and after the question closing date. [At] the end I discovered the even through some blogger who uploaded pictures of scaffoldings next to one of the targeted monuments.
Conversation with Cmeinel #2 Forecaster on the HFC Leaderboard
1. Do you have any thoughts on if you learned anything from the HFC Challenge (about forecasting, about world events, etc.)?
I accepted an invitation to join a Slack discussion group on GJ Open that now has eleven people. We turned out to be a good mix as we have complementary skills. Some are better at modeling, some better at gathering information, some at analysis. I owe much of my standing on the HFC leaderboard to their assistance. I have been amazed at how many various and complementary strategies my teammates have revealed to me. Right now, ten out of eleven of us are on one or more leaderboards. So, what we are doing, works.
2. Were there any interesting questions that stood out – maybe a particularly challenging one from the HFC Challenge? If so, how did you tackle it?
The questions on lethal violence. These questions were outstanding in that they included information on where we could obtain base rate data. So, for each question, step one was to populate a spreadsheet. But when I saw how volatile these base rates were, at first I regretted competing. But then I used Twitter to locate people on the ground for each of the nations involved and shared their reports both on GJ Open and with our team. Others in our Slack team found additional sources of breaking news. Sharing all this information, and discussing what it all meant, we did pretty well.
What I liked best about tackling lethal violence problems is that I also discovered the good sides of the people of these nations. I was particularly impressed by the kindness, optimism and grit of the people of Somalia. I now have more hope for the world.