{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T11:18:58Z","timestamp":1774523938908,"version":"3.50.1"},"reference-count":0,"publisher":"IOS Press","isbn-type":[{"value":"9781643686387","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,12,2]],"date-time":"2025-12-02T00:00:00Z","timestamp":1764633600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,12,2]]},"abstract":"<jats:p>Large language models (LLMs) with extended context windows show promise for complex legal reasoning tasks, yet their ability to understand long legal documents remains insufficiently evaluated. Developing long-context benchmarks that capture realistic, high-stakes tasks remains a significant challenge in the field, as most existing evaluations rely on simplified synthetic tasks that fail to represent the complexity of real-world document understanding. Overruling relationships are foundational to common-law doctrine and commonly found in judicial opinions. They provide a focused and important testbed for long-document legal understanding that closely resembles what legal professionals actually do. We present an assessment of state-of-the-art LLMs on identifying overruling relationships from U.S. Supreme Court cases using a dataset of 236 case pairs. Our evaluation reveals three critical limitations: (1) era sensitivity \u2013 the models show degraded performance on historical cases compared to modern ones, revealing fundamental temporal bias in their training; (2) shallow reasoning \u2013 models rely on shallow logical heuristics rather than deep legal comprehension; and (3) context-dependent reasoning failures \u2013 models produce temporally impossible relationships in complex open-ended tasks despite maintaining basic temporal awareness in simple contexts. Our work contributes a benchmark that addresses the critical gap in realistic long-context evaluation, providing an environment that mirrors the complexity and stakes of actual legal reasoning tasks. The full dataset can be accessed at https:\/\/github.com\/lizhang-AIandLaw\/Do-LLMs-Truly-Understand-When-a-Precedent-Is-Overruled<\/jats:p>","DOI":"10.3233\/faia251592","type":"book-chapter","created":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T12:04:56Z","timestamp":1764849896000},"source":"Crossref","is-referenced-by-count":1,"title":["Do LLMs Truly \u201cUnderstand\u201d when a Precedent Is Overruled?"],"prefix":"10.3233","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0375-1793","authenticated-orcid":false,"given":"Li","family":"Zhang","sequence":"first","affiliation":[{"name":"University of Pittsburgh"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3674-5456","authenticated-orcid":false,"given":"Jaromir","family":"Savelka","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5535-0759","authenticated-orcid":false,"given":"Kevin","family":"Ashley","sequence":"additional","affiliation":[{"name":"University of Pittsburgh"}]}],"member":"7437","container-title":["Frontiers in Artificial Intelligence and Applications","Legal Knowledge and Information Systems"],"original-title":[],"link":[{"URL":"https:\/\/ebooks.iospress.nl\/pdf\/doi\/10.3233\/FAIA251592","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T12:04:57Z","timestamp":1764849897000},"score":1,"resource":{"primary":{"URL":"https:\/\/ebooks.iospress.nl\/doi\/10.3233\/FAIA251592"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,2]]},"ISBN":["9781643686387"],"references-count":0,"URL":"https:\/\/doi.org\/10.3233\/faia251592","relation":{},"ISSN":["0922-6389","1879-8314"],"issn-type":[{"value":"0922-6389","type":"print"},{"value":"1879-8314","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,2]]}}}