When AI Becomes a False Prophet: A Cautionary Tale for Forecasters

Written by

When AI Becomes a False Prophet: A Cautionary Tale for Forecasters

With a nod to Taylor Swift and Travis Kelce, Superforecaster Ryan Adler discusses the gospel according to AI and why forecasters should always verify their sources.

Google’s AI Overview references an AI-generated video to support a false claim.

The promises of artificial intelligence have set up camp in media headlines over the past few years. ChatGPT has become a household name, billions are being spent just to power the equipment to run these programs and models, and the cutting-edge technology is front and center in ongoing tensions between the US and China. It hasn’t left any aspect of human activity untouched, including forecasting.

To be sure, the impacts already felt cannot be understated. We are looking at the front end in what I’m confident will be a seismic shift in society, with large swaths of labor markets around the globe being shaken to their core. That said, we aren’t there yet.

Here’s a recent example of how AI took itself out at the knees regarding a recent forecasting question on Good Judgment Open. In late April 2025, the time came to close a question regarding potential nuptials between Kansas City Chiefs star Travis Kelce and pop superstar Taylor Swift: “Before 19 April 2025, will Travis Kelce and Taylor Swift announce or acknowledge that they are engaged to be married?” (It’s not my favorite subject matter, but we try to maintain a diverse pool of questions.)

As a moderately rabid Chiefs fan myself, I was confident the answer was no, because that would have made headlines across media outlets. However, a key part of the job of running a forecasting platform is being in the habit of double and triple checking. So, I checked with Google. I entered “Are Travis Kelce and…” into the search field, which immediately autofilled to “are travis and taylor engaged?” (The first-name thing with pop culture stars annoys me to no end, but I digress.) To my surprise, Google’s AI preview popped up immediately.

“Yes, according to reports, Travis Kelce and Taylor Swift are engaged.”

“Trust, but verify”

Skeptical, I looked at what the experimental generative AI response was using as a reference to return such a statement. That’s when things got fun.

The first link of the cited material was a YouTube video. Keep in mind that Google, the search engine I used to start my research, owns YouTube. The account that posted the video? DangerousAI. That alone raises more red flags than a May Day parade in Moscow circa 1974. The brief video, dated 24 February 2025, purported to show Travis Kelce announcing that Swift and he “got engaged last week.” However, as the video progressed, the absurdity of Kelce’s putative announcement became perfectly clear.

To sum up, Google’s AI system linked to search was fooled by an AI product posted on another Google platform to give a patently false response.

I don’t highlight this incident as a criticism of Google. However, it should serve as a warning. I’ve seen some GJ Open forecasters take AI responses as gospel. I’m here to tell you that in matters of facts vs fiction, AI is very capable of being a false prophet. This is not to say that AI isn’t an incredibly valuable tool. It certainly is! We are finding more and more uses for it at Good Judgment, but we put it through its paces long before we deem it reliable for a particular role. As the Russian proverb instructs, “Trust, but verify.” (No, President Reagan didn’t say it first.) When it comes to AI and everything else you see online, my suggestion is that you just verify.

Do you have what it takes to be a Superforecaster? Find out on GJ Open!

* Ryan Adler is a Superforecaster, GJ managing director, and leader of Good Judgment’s question team

Sneak Peek: How FutureFirst Superforecast Alerts Keep You Ahead

Written by

Sneak Peek: How FutureFirst™ Superforecast Alerts Keep You Ahead

 

Subscribers to FutureFirst™, Good Judgment’s exclusive forecasting and monitoring service, enjoy 24/7 access to expert-driven insights on key economic, geopolitical, and market-moving questions. Superforecast Alerts, included with the service, provide timely updates when:

  • A high-profile question is added to the platform.
  • The aggregate forecast shifts significantly, signaling a meaningful change in Superforecasters’ expectations.

Related questions are grouped into clusters to offer broader analytical context. See our discussion on the relevance-rigor trade-off for more details.

These alerts empower decision-makers with actionable intelligence provided by Good Judgment’s elite Superforecasters—a diverse global team with a proven track record of accuracy. To illustrate the value of these alerts, here’s an example available to FutureFirst subscribers on 18 January 2025. This alert focused on expected interest rate decisions from the European Central Bank (ECB) and the Bank of England (BoE).

Superforecasting Central Banks

What will be the European Central Bank’s (ECB) most recently announced “Deposit facility” interest rate as of January 30, 2025?
Date Less than 2.50% At least 2.50%, but less than 2.75% At least 2.75%, but less than 3.00% At least 3.00%, but less than 3.25% 3.25% or higher
18 Jan 2025 0% 0% 94% 6% 0%
What will be the Bank of England’s Bank Rate at the close of business on February 6, 2025?
Date Less than 4.25% At least 4.25%, but less than 4.50% At least 4.50%, but less than 4.75% At least 4.75%, but less than 5.00% 5.00% or more
18 Jan 2025 0% 0% 74% 26% 0%

 
18 Jan 2025 14:57 ET – To kick off 2025, Superforecasters anticipate the following interest rate decisions. They forecast a 94% probability that the European Central Bank will lower its “Deposit facility” interest rate on 30 January 2025. They expect it to fall within a range of 2.75% to less than 3.00%, with only a minimal chance that the rate will remain the same. Economists in a Reuters survey unanimously expect a 25-basis-point reduction, reflecting the ECB’s ongoing easing cycle amid Europe’s sluggish economic growth and moderately increasing inflation rates, such as a recent 2.4% increase in the eurozone. Despite a gradual rise in inflation, policymakers aim to meet a 2% goal by mid-2025, facing pressures from rising services prices and low unemployment. Yet, concerns about energy costs and inflationary pressures in certain eurozone areas could challenge further rate cuts.

Looking ahead to the Bank of England’s meeting on 6 February 2025, Superforecasters see a 74% probability that the Bank Rate will decrease to at least 4.50% but less than 4.75%, with a 26% probability for it to stay the same at a range of at least 4.75% but less than 5.00%. A recent dip in UK inflation to 2.5% has strengthened expectations for a rate cut, as economists see evidence of the restrictive monetary policy’s effectiveness in easing inflation pressures. However, ongoing concerns about inflationary pressures driven by the government’s tax measures and rising gilt yields may complicate the central bank’s easing strategies. Despite these uncertainties, most surveyed economists anticipate a quarter-point cut in the upcoming meeting.

Superforecaster Quotes

  • (European Central Bank): “The ECB is two weeks away from what’s widely expected to be the fifth rate cut of this easing cycle. Despite inflation ticking higher last month, policymakers are confident of meeting their 2% goal in 2025 and remain concerned about Europe’s sluggish economy.”
  • (European Central Bank): “The European Central Bank can ease policy further this year but must find a middle ground that neither induces a recession nor causes an undue delay in curbing inflation, ECB chief economist Philip Lane told an Austrian newspaper.”
  • (European Central Bank): “If interest rates fall too quickly, it will be difficult to bring services inflation under control…But we also don’t want rates to remain too high for too long, because that would weaken the inflation momentum in such a way that the disinflation process would not stop at 2% but inflation could materially fall below target.”
  • (Bank of England): “Inflation numbers better than expected, making another cut more likely.”
  • (Bank of England): “The deceleration in inflation, which still remains above the BoE 2% target, is likely to be taken by MPC members as a sign that the restrictive monetary policy stance is helping to bring inflation closer to the target. As such, it will give MPC members enough evidence to cut rates by 25bps at their next meeting in February.”
  • (Bank of England): “Despite unpredictability around the future policies of US President Elect Donald Trump and their inflationary impact, the economy’s weak momentum might confirm a scenario in which the Bank of England proceeds with faster interest rate cuts in 2025.”

 
Superforecaster Sources

 

Why FutureFirst?

Superforecast Alerts like this give our clients a decisive advantage by providing early warnings and Superforecasters’ interpretations of emerging trends. Whether you’re monitoring central bank decisions, geopolitical events, or financial markets, FutureFirst keeps you informed—so you can make smarter, more confident decisions ahead of the curve.

📩 Want to receive exclusive alerts like this? Subscribe to FutureFirst today.

Good Judgment’s 2024 in Review

Written by

Good Judgment’s 2024 in Review

Superforecasters always keep score. As we turn the page to 2025 at Good Judgment Inc, we look back at 2024 for highlights, statistics, and key developments.

What a year it was! From monetary policy shifts to volatile election outcomes, our Superforecasters tackled some of the most complex forecasting questions to date. Good Judgment’s clients and FutureFirst™ subscribers posed 151 questions to the Superforecasters on our proprietary platform, with a total of 1,132 forecasting questions live in 2024 across our platforms, including on GJ Open, our public forecasting site.

Forecasting Highlights

Good Judgment’s Superforecasters continued to outperform financial markets in 2024. This caught the eye of, among others, Financial Times reporters.

“A group of lay people with a talent for forecasting have consistently outperformed financial markets in predicting the Fed’s next move,” wrote data journalist Joel Suss for FT’s Monetary Policy Radar in July 2024. For the year as a whole, the Superforecasters outperformed the futures markets by 22%.

Our partnership with ForgeFront and the UK Government’s Futures Procurement Network continued to grow in 2024. In November, Good Judgment’s CEO Dr. Warren Hatch spoke remotely at the Department for Environment, Food and Rural Affairs (DEFRA) Futures Trend Briefing, discussing the role of Superforecasting in supporting the UK’s Biological Security Strategy.

We also saw an increased interest in subscriptions to specialized forecast channels on FutureFirst, including the “Superforecasting the Middle East” and “Superforecasting US Politics” channels. Individual FutureFirst channels provide access to a range of forecasting questions grouped by theme or topical focus and, like the all-inclusive subscription option, come with daily forecast updates and API access.

Media and Research

Throughout the past year, Good Judgment continued partnering with leading media organizations. Notably, The Economist once again featured our Superforecasters’ outlook in “The World Ahead 2025.” The focus for this year was on US tariffs, elections in Germany, Canada, and Australia, China’s inflation, and the Russia-Ukraine conflict. Our Superforecasters were also referenced in The New York Times, Wired, and Vox, among other media outlets. (See our Press page for the full list.)

Our research partners at the Forecasting Research Institute continued exploring the applications of Superforecasting in approaching key questions, from AI forecasting capabilities to nuclear risk. The institute’s ForecastBench, released on 1 October 2024, revealed that the top-performing large language models (LLMs) lagged behind Superforecasters by 19%. (See our Science of Superforecasting page for all relevant scientific publications.)

Private and Public Challenges on GJ Open

GJ Open continued partnering with media, businesses, and educational institutions to host public and private forecasting challenges on their behalf. Our public challenges included those from UBS Asset Management Investments, Man Group, Fujitsu, City University of Hong Kong, Harvard Kennedy School, and The Economist. We also hosted private challenges for our partners to help them identify in-house forecasting talent and train interns and staff in probabilistic thinking and accountable decision-making. We supplied the tools, a secure platform, data analysis, and Superforecasters’ feedback.

Training

As we continued to provide forecasting training to professional teams and individuals, we saw an increase in virtual as well as in-person workshops, seminars, and presentations, which were conducted for hundreds of individuals in dozens of government, non-profit, and private-sector organizations across the United States and abroad, including in the UK, the Netherlands, and Turkey.

We’re excited to continue this journey in 2025 and wish everyone a Happy New Year. May it be a year of thoughtful forecasts and better decisions!