The Crowd May Have Been More Right Than 55% Might Look

Posted on Posted in GJ Open, Uncategorized


Ryan Adler is a Superforecaster and Senior Consultant for Good Judgment who specializes in legal analysis for Good Judgment’s question writing team. He also administers the SCOTUS Challenge on Good Judgment Open, a collaboration between SCOTUSblog and Good Judgment that facilitates probabilistic thinking about court decisions. Ryan can be reached at

The Court was kind enough to kick off June by handing down its decision in Masterpiece Cakeshop, Ltd. v. Colorado Civil Rights Commission, easily one of the most politically and socially-charged cases of this term.  At first blush, it doesn’t appear close.  Seven of nine Justices ruled to reverse the Colorado Court of Appeals, stating that shop owner Jack Phillips’ rights under the Free Exercise Clause of the First Amendment had been violated.  However, the reality was much closer.  Justice Kennedy’s opinion for the Court was careful to review the procedural history of the complaint.  The result was a totality of the circumstances test in everything but name.

The real battle, in this Court watcher’s humble opinion, was the duel of concurrences between Kagan and Gorsuch.  Justice Kagan reads the opinion of the court as narrow in the sense that but for the animus displayed by the Commission, it could have decided against Mr. Phillips if it had been differently handled.  Conversely, Justice Gorsuch casts the ruling in a broader light.  So what does that mean for 431 forecasters and their nearly 700 forecasts on the outcome of this case?


Cakeshop Forecast Graphic

For starters, it’s worth noting the initial consensus prior to oral arguments at 25% through mid-November, with a slow walk up to 55% by December 7th, the date of arguments.  Nothing new happened in this time frame, save for an increase in the number of forecasters, which increased from 129 to 187 in the three weeks prior to arguments.  From there, with things seemingly going well for the petitioners, we saw an increase in consensus volatility for the next few months, hitting as high as 70%.  However, by February 21st and our 360th forecaster, we were back to 55% and effectively stayed there for the rest of the case.

To quote Admiral Greer, the data support no conclusions as yet, but I will offer two hypotheses.  First, the SCOTUS Challenge was in its infancy in the early weeks of this question, and word was still getting around.  With the passage of time and the approach of oral arguments, forecaster engagement (individually and in the aggregate) ramped up as arguments were to be held.  As radio silence set in after arguments, forecasters (the count for which nearly doubled until the retrenchment at 55%) sifted through pontificating at all levels and found themselves back where they nearly began.

Second, I am wondering if the seemingly low but accurate probability reflected, at least in part, the relatively narrow scope of the eventual ruling.  This was no slam dunk producing a new, categorical line of defense in the name of the Free Exercise Clause.  So, in other words, the crowd called it close in quantitative terms simply because it would turn out to be so close in qualitative terms.  Again, these are mere hypotheses, so don’t run with them.

Share on FacebookTweet about this on TwitterShare on LinkedInEmail this to someone

Leave a Reply

Your email address will not be published. Required fields are marked *