Research

Publications

• “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models.” 2024. Journal of Legal Analysis 16(1): 64-93 (with Varun Magesh, Mirac Suzgun, and Daniel E. Ho). We show that popular AI tools like ChatGPT frequently “hallucinate,” or invent, false information about American case law, calling into question their ability to democratize access to justice.
  • Media: Bloomberg, NPR/Marketplace, The Hill, The Register, Fast Company, Scientific American, New York Times
  • Judicial citations: Supreme Court of British Columbia (Canada)
  • [Replication code]
  • [Benchmarking dataset]
• “Chain Novel, or Markov Chain? Estimating the Authority of U.S. Supreme Court Case Law.” 2024. Journal of Empirical Legal Studies 21(4): 861-898. I show how modeling the network of U.S. Supreme Court case law as a Markov chain unlocks an intuitive estimator of case authority that outperforms existing approaches in a variety of validation tasks.
  • [Replication code]
  • [HolmesRank scores]
  • [Depth of treatment data]
• “The Dogma Within? Examining Religious Bias in Private Title VII Claims.” 2021. Journal of Empirical Legal Studies 18(4): 742-764 (with Devan N. Patel and Matthew E.K. Hall). Examining an original dataset of private discrimination claims, we debunk the popular belief that judges’ religions cause them to decide cases in a distinctive way.
• “Eyecite: A Tool for Parsing Legal Citations.” 2021. Journal of Open Source Software 6(66): 3617 (with Jack Cushman and Michael Lissner). We open-source a highly performant engine for the analysis of legal citations, trained and deployed on the datasets of the Free Law Project and the Caselaw Access Project.
  • [Python package]

Working papers

• “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.” (with Varun Magesh, Faiz Surani, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho). We demonstrate the distinctive challenges that retrieval-agumented generation (RAG) faces in the legal domain, stressing the need for transparent benchmarking of commercially available legal AI products.
  • Media: Fortune, Wired, VentureBeat, Law360, Law.com, Gizmodo
• “Natural Language Evidence for the Existence and Effect of Jurisprudential Regimes.” (with Matthew E.K. Hall). Tracing the linguistic development of legal doctrine over time, we present new evidence for the way that law constrains the voting behavior of Supreme Court justices.
• “Does Theory Matter? Testing the Influence of Constitutional Theory on Specific and Diffuse Support for the U.S. Supreme Court.” (with Matthew E.K. Hall). Invoking some of the leading scholars of normative constitutional theory (Dworkin, Posner, Barnett, etc.), we test their implicit empirical claims in a national survey experiment.

Other writing

• “Why did Sen. Graham grill Ketanji Brown Jackson about her religious faith?” The Washington Post. March 2022.
• “Which of Trump’s Supreme Court choices might be most reliably conservative?” The Conversation. September 2020.