Decoding SCOTUS: Navigating Media Bias in Supreme Court Forecasting

Superforecaster, GJ managing director, and leader of Good Judgment’s question team, Ryan Adler shares tips on how to approach forecasting Supreme Court decisions.

May is a lovely time of year. Lots of daylight, newborn critters all around, and Supreme Court junkies start to get their fix from now through the end of June. As someone who has watched the Court enthusiastically for more than a quarter of a century, I always look forward to decision season. However, in all those years, I’ve learned a painful lesson. The press is generally terrible at reporting about the Court. For Court junkies like me, it’s not that big of a deal. I learned early in my studies that if I want to know what a case means, I need to dig into the weeds myself. But as someone who has also written dozens of SCOTUS forecasting questions, I know that many forecasters don’t have the benefit of having read thousands of pages of appellate decisions to instruct their analysis. So, as is the case for anybody forecasting on a topic with which they aren’t intimately familiar, they lean on press reporting to fill in the gaps.

“Creating an emotional investment in the outcome of a case is poison for a forecaster.”

Alas, most journalists and talking heads intent to tell readers and viewers “how it is” end up just regurgitating heavily politicized schtick from one side of the aisle or the other. A glaring example is Trump v. United States, the presidential immunity case out of the DC Circuit Court of Appeals. For the vast majority of the press, it’s a question of what the Court will let happen to Trump in Jack Smith’s federal election interference case. For anyone who understands the Constitution and listened to oral arguments in the case, the Court knows it’s grappling with something much bigger and fundamental than Trump’s actions and statements on January 6th. Questions of presidential immunity go to the heart of the structure of the US’ separation of powers, perhaps the greatest innovation from the Founding Fathers that has proven key to good governance in modernity. Framing the case as democracy in peril (you can find such arguments from both Trump’s supporters and detractors) is, in this writer’s opinion, theater. This isn’t to say that the events of January 6th are nothing of concern, nor does it make light of the potential precedents to be set. But let’s be honest: if the party affiliation of the defendant in this case were flipped, who honestly thinks that the arguments from the “legal experts” in the media wouldn’t look far different? That doesn’t make these people liars, but it highlights that these “experts” are doing something very different than what we ask forecasters to do—figure out what will happen irrespective of what they feel should happen. Such “experts” placate and/or infuriate, which is how politics works.

What is to be done?

Creating an emotional investment in the outcome of a case is poison for a forecaster. We can debate all day about the political implications of a Court decision, and the nine robed ones don’t operate in a vacuum. That all notwithstanding, there is a stark difference between being mindful of political factors that influence the conversation among them and acting as though those implications represent the sole axis upon which the case will turn. It may seem hard not to see anything and everything through a political lens, and most media outlets have given up pretending that their reporting isn’t first run through a political prism.

So, what is to be done? As James Madison stated that ambition must be made to counteract ambition, this writer suggests that cynicism must be made to counteract cynicism. Take the words of any legal “expert” or talking head with a large grain of salt. Being a lawyer doesn’t make you an expert on the Constitution. Many lawyers who practice would probably admit that the last time they paid attention to constitutional law was prepping for a bar exam. That fact is not mitigated by gaining the title of “contributor.” If what you read or hear feels like it’s what you may personally want to read or hear, be suspicious. That’s sound advice for anything in the media, but it’s especially crucial for navigating the flood of gibberish that inundates the airwaves and web when a high-profile case makes it to Washington.

Do you have what it takes to be a Superforecaster? Find out on GJ Open!

 

Tribute to Daniel Kahneman: Insights and Memories from Superforecasters

 

---

“Superforecasters are better than others at finding relevant information—possibly because they are smarter, more motivated, and more experienced at making these kinds of forecasts… Superforecasters are less noisy than…even trained teams.”

—Kahneman, Sibony, & Sunstein, in Noise: A Flaw in Human Judgment

“For as long as I have been running forecasting tournaments, I have been talking with Daniel Kahneman about my work. For that, I have been supremely fortunate. … Talking to Kahneman can be a Socratic experience: energizing as long as you don’t hunker down into a defensive crouch. So in the summer of 2014, when it was clear that superforecasters were not merely superlucky, Kahneman cut to the chase: ‘Do you see them as different kinds of people, or as people who do different kinds of things?’ My answer was, ‘A bit of both.’”

—Phil Tetlock in Superforecasting: The Art and Science of Prediction

---

 

Daniel Kahneman, Nobel Prize winner and a friend of Good Judgment, died aged 90 on 27 March 2024. He was a true pioneer in the realms of judgment, decision-making, and the psychological underpinnings that affect our forecasting work. Beyond his monumental contributions to psychology and economics, Kahneman took a keen interest in the art and science of Superforecasting, discussing Superforecasters in his last published book Noise and gracing us with his presence at a Good Judgment workshop. In this tribute to Daniel Kahneman, Superforecasters share insights and memories on how his work influenced and enriched them.

Superforecaster Dan Mayland:

The best tribute I can give is to share a photo of my copy of Thinking, Fast and Slow, which I read around the same time I started forecasting. I have this (slightly!) obsessive habit of affixing small Post-It notes to book passages that I think are particularly insightful and don’t want to lose track of once I turn the page. (Even though, the mind being what it is, I do.) It’s my version of taking pictures of sunsets in an attempt to preserve the memory. The photo below speaks to the level of insight I thought Kahneman had to offer.

 

Co-founder and CEO Emeritus of Good Judgment Inc, Terry Murray:

In the early days of Good Judgment Inc, I had the privilege of being in an extended conversation that included our co-founders Phil Tetlock and Barb Mellers along with Danny Kahneman and a couple of his colleagues. I expected Danny to have great questions and insights on the substantive side of our forecasting work because of his own research interests and his past interest in the Good Judgment Project itself. What surprised me most at the time was that Danny’s questions about the business side of our fledgling company were among the best questions anyone had ever posed to me.

Looking back now, I shouldn’t have been surprised. The handful of opportunities I had to interact directly with Danny all confirmed what his writings on judgment and decision-making tell us: Danny was a master at cutting to the chase. Whatever the topic under discussion, he would quickly filter out the noise and hone in on the signal. If I could create the ideal AI personal assistant, it would be one that would pose Danny-like questions to challenge me until I too filtered out the noise and focused on what is most important. I’d like to think that Danny would approve.

Superforecaster and CEO of Good Judgment Inc, Warren Hatch:

A while back, we ran a “Noise Challenge” on Good Judgment Open that was inspired by the launch of the book Daniel Kahneman co-wrote. He generously agreed to sign copies that we could present to the top performers at the end of the competition. So I went by his house to meet him in the lobby. I figured it would take five minutes of his time tops. Instead, we sat down and spent the better part of an hour going over some of the questions then posed to the Superforecasters. It was a masterclass.

Superforecaster Giovanni Ciriani:

I read Thinking, Fast and Slow in early 2013; on 21 March 2013, David Brooks wrote the NY Times column “Forecasting Fox,” which was centered on Tetlock and the Good Judgment Project. It credited Kahneman’s Thinking, Fast and Slow as training material. I read the column uniquely because the book was mentioned. On 22 March 2013, I signed up for the GJP. Of course, one could play counterfactuals, to see which factor was more important in my joining, but I see Kahneman’s as the sine qua non, the condition without which it would not have happened.

Superforecaster and GJ Director Ryan Adler:

I will ashamedly confess that my knowledge of Danny Kahneman’s work was limited before I found myself in the Good Judgment Project, but immersing myself in it was and is an absolute pleasure. To borrow some phrasing from Robert Heinlein, Kahneman and Tversky studied what man is, not what they wanted him to be, and it is saddening to know that there will be less of that kind of thinking with his passing.

Superforecaster Vijay Karthik:

I have learnt a lot by reading articles regarding him and through his book. Life has become a lot more enriched trying to put his recommendations into action, to the best extent possible.

Superforecaster David Fisher:

I remember listening to him on a podcast. He was pessimistic that information alone could change opinions about global warming. He said only if someone identified with the person giving the information were they likely to consider it. I have read Thinking, Fast and Slow and most of Noise and have nothing but respect. Decisions only made because of someone’s economic interests? He disproved that. Deserved the Nobel.

Superforecaster Dwight Smith:

His influence on my life has been tremendous. It was through his work that I became a far better forecaster. And that has led to a multitude of adventures and good works. To be so profoundly influenced by such a great mind even once in a lifetime is quite a gift.