{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T11:11:01Z","timestamp":1785409861163,"version":"3.56.0"},"reference-count":78,"publisher":"National Academy of Sciences","issue":"30","license":[{"start":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T00:00:00Z","timestamp":1784505600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2026,7,28]]},"abstract":"<jats:p>Despite substantial excitement around the use of AI in law, little information exists on the performance and associated risks of the domain\u2019s widely marketed tools. Recent work, for instance, has demonstrated the significant potential for \u201challucinations\u201d\u2014wherein models make up facts, law, and precedent\u2014leading Chief Justice Roberts to spotlight this risk in his annual report on the judiciary. We argue that there is a need for public AI benchmarking in law. First, relative to other AI application domains, the legal AI ecosystem lacks legibility\u2014there is little information about the design and performance of many commercial legal AI systems. Legal AI has not benefited from the types of benchmarking that have catalyzed, measured, and informed AI innovation and responsible use in other domains. Second, we articulate the challenges of the institutional design of benchmarking. We illustrate how benchmarks can be captured, watered down, and abused. Careful institutional design around the why, who, what, and how of benchmarking will be critical to navigate difficult tradeoffs of transparency, objectivity, expertise, and resources. Third, addressing legal AI\u2019s illegibility requires matching institutional models to available resources and constraints. Rather than advocating for a single \u201cbest\u201d approach to benchmarking, we show how benchmarking strategies depend on available resources.<\/jats:p>","DOI":"10.1073\/pnas.2509757122","type":"journal-article","created":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T19:15:07Z","timestamp":1784574907000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":1,"title":["There is no free benchmark: An institutional view of legal AI benchmarking"],"prefix":"10.1073","volume":"123","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-5120-1726","authenticated-orcid":false,"given":"Neel","family":"Guha","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Law School, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Computer Science, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Regulation, Evaluation, and Governance Lab (RegLab), Stanford University","place":["Stanford"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andy K.","family":"Zhang","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Law School, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Computer Science, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Regulation, Evaluation, and Governance Lab (RegLab), Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/01an7q238","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Electrical Engineering and Computer Sciences, University of California","place":["Berkeley"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christine","family":"Tsang","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Regulation, Evaluation, and Governance Lab (RegLab), Stanford University","place":["Stanford"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6155-649X","authenticated-orcid":false,"given":"Christopher D.","family":"Manning","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Computer Science, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Linguistics, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University","place":["Stanford"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7121-5696","authenticated-orcid":false,"given":"Julian","family":"Nyarko","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Law School, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Legal Innovation through Frontier Technology Lab (liftlab), Stanford University","place":["Stanford"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2195-5469","authenticated-orcid":false,"given":"Daniel E.","family":"Ho","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Law School, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Computer Science, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Regulation, Evaluation, and Governance Lab (RegLab), Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Political Science, Stanford University","place":["Stanford"]},{"id":[{"id":"https:\/\/ror.org\/00f54p054","id-type":"ROR","asserted-by":"publisher"}],"name":"Stanford Institute for Economic Policy Research (SIEPR), Stanford University","place":["Stanford"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2026,7,20]]},"reference":[{"key":"e_1_3_4_1_2","unstructured":"R. Bommasani On the opportunities and risks of foundation models. arXiv [Preprint] (2021). http:\/\/arxiv.org\/abs\/2108.07258 (Accessed 3 September 2025)."},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2023.0254"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-1915"},{"key":"e_1_3_4_4_2","unstructured":"A. Garg M. Ma Opportunities and challenges in legal AI (2025). https:\/\/law.stanford.edu\/wp-content\/uploads\/2025\/01\/Legal-AI-Opportunities-and-Challenges-Whitepaper.pdf. Accessed 3 September 2025."},{"key":"e_1_3_4_5_2","unstructured":"J. John G. Roberts 2023 year-end report on the federal judiciary 2023; https:\/\/www.supremecourt.gov\/publicinfo\/year-end\/2023year-endreport.pdf."},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1111\/jels.12413"},{"key":"e_1_3_4_7_2","doi-asserted-by":"crossref","unstructured":"L. L. Ouellette A. Motomura J. Reinecke J. S. Masur Can AI hold office hours? Available at SSRN 5166938 2025; https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=5166938.","DOI":"10.2139\/ssrn.5166938"},{"key":"e_1_3_4_8_2","unstructured":"A. Salinas A. Haim J. Nyarko What\u2019s in a name? Auditing large language models for race and gender bias. arXiv [Preprint] (2024). http:\/\/arxiv.org\/abs\/2402.14875 (Accessed 3 September 2025)."},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1093\/jla\/laae003"},{"key":"e_1_3_4_10_2","unstructured":"D. Charlotin AI hallucination cases (2025). https:\/\/www.damiencharlotin.com\/hallucinations\/. Accessed 3 September 2025."},{"key":"e_1_3_4_11_2","unstructured":"P. Henderson AI law tracker (2025). https:\/\/www.polarislab.org\/ai-law-tracker.html. Accessed 3 September 2025."},{"key":"e_1_3_4_12_2","unstructured":"M. Poggio Gen AI shows promise - and peril - for pro se litigants (2024). https:\/\/www.law360.com\/articles\/1812918\/gen-ai-shows-promise-and-peril-for-pro-se-litigants. Accessed 3 September 2025."},{"key":"e_1_3_4_13_2","first-page":"1","article-title":"Promises and pitfalls of artificial intelligence for legal applications","volume":"2","author":"Kapoor S.","year":"2024","unstructured":"S. Kapoor, P. Henderson, A. Narayanan, Promises and pitfalls of artificial intelligence for legal applications. J. Cross Discip. Res. Comput. Law 2, 1\u201313 (2024).","journal-title":"J. Cross Discip. Res. Comput. Law"},{"key":"e_1_3_4_14_2","unstructured":"P. Liang Holistic evaluation of language models (Featured Certification Expert Certification Outstanding Certification) (2023)."},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","unstructured":"N. Guha J. Nyarko D. E. Ho C. R\u00e9 \u201cBuilding GenAI benchmarks: A case study in legal applications\u201d in The Oxford Handbook of the Foundations and Regulation of Generative AI P. Hacker A. Engel S. Hammer B. Mittelstadt Eds. (Oxford University Press 2025).","DOI":"10.1093\/oxfordhb\/9780198940272.013.0007"},{"key":"e_1_3_4_16_2","unstructured":"C. E. Jimenez \u201cSWE-bench: Can language models resolve real-world github issues?\u201d in The Twelfth International Conference on Learning Representations (2024)."},{"key":"e_1_3_4_17_2","unstructured":"J. Buolamwini T. Gebru \u201cGender shades: Intersectional accuracy disparities in commercial gender classification\u201d in Conference on Fairness Accountability and Transparency (PMLR 2018) pp. 77\u201391."},{"key":"e_1_3_4_18_2","doi-asserted-by":"crossref","unstructured":"X. Wang \u201cChestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017) pp. 2097\u20132106.","DOI":"10.1109\/CVPR.2017.369"},{"key":"e_1_3_4_19_2","doi-asserted-by":"crossref","unstructured":"L. Zheng \u201cA reasoning-focused legal retrieval benchmark\u201d in Proceedings of the 2025 Symposium on Computer Science and Law (2025) pp. 169\u2013193.","DOI":"10.1145\/3709025.3712219"},{"key":"e_1_3_4_20_2","doi-asserted-by":"crossref","unstructured":"I. Chalkidis \u201cLexGLUE: A benchmark dataset for legal language understanding in English\u201d in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) S. Muresan P. Nakov A. Villavicencio Eds. (Association for Computational Linguistics Dublin Ireland 2022) pp. 4310\u20134330.","DOI":"10.18653\/v1\/2022.acl-long.297"},{"key":"e_1_3_4_21_2","first-page":"1473","article-title":"Ai regulation has its own alignment problem: The technical and institutional feasibility of disclosure, registration, licensing, and auditing","volume":"92","author":"Guha N.","year":"2024","unstructured":"N. Guha , Ai regulation has its own alignment problem: The technical and institutional feasibility of disclosure, registration, licensing, and auditing. Geo. Wash. L. Rev. 92, 1473 (2024).","journal-title":"Geo. Wash. L. Rev."},{"key":"e_1_3_4_22_2","first-page":"117","article-title":"An institutional view of algorithmic impact assessments","volume":"35","author":"Selbst A. D.","year":"2021","unstructured":"A. D. Selbst, An institutional view of algorithmic impact assessments. Harv. JL Tech. 35, 117 (2021).","journal-title":"Harv. JL Tech."},{"key":"e_1_3_4_23_2","doi-asserted-by":"crossref","unstructured":"I. D. Raji P. Xu C. Honigsberg D. Ho \u201cOutsider oversight: Designing a third party audit ecosystem for AI governance\u201d in Proceedings of the 2022 AAAI\/ACM Conference on AI Ethics and Society (2022) pp. 557\u2013571.","DOI":"10.1145\/3514094.3534181"},{"key":"e_1_3_4_24_2","doi-asserted-by":"crossref","unstructured":"L. Zheng N. Guha B. R. Anderson P. Henderson D. E. Ho \u201cWhen does pretraining help? Assessing self-supervised learning for law and the casehold dataset of 53 000+ legal holdings\u201d in Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law (2021) pp. 159\u2013168.","DOI":"10.1145\/3462757.3466088"},{"key":"e_1_3_4_25_2","first-page":"387","article-title":"ChatGPT goes to law school","volume":"71","author":"Choi J. H.","year":"2021","unstructured":"J. H. Choi, K. E. Hickman, A. B. Monahan, D. Schwarcz, ChatGPT goes to law school. J. Leg. Educ. 71, 387 (2021).","journal-title":"J. Leg. Educ."},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2023.0159"},{"key":"e_1_3_4_27_2","unstructured":"A. Haupt E. Brynjolfsson \u201cPosition: AI should not be an imitation game: Centaur evaluations\u201d in Forty-second International Conference on Machine Learning Position Paper Track (2025)."},{"key":"e_1_3_4_28_2","unstructured":"K. Ethayarajh Y. Choi S. Swayamdipta \u201cUnderstanding dataset difficulty with v-usable information\u201d in International Conference on Machine Learning (PMLR 2022) pp. 5988\u20136008."},{"key":"e_1_3_4_29_2","unstructured":"H. de Vries D. Bahdanau C. D. Manning Towards ecologically valid research on language user interfaces. arXiv [Preprint] (2020). http:\/\/arxiv.org\/abs\/2007.14435 (Accessed 3 September 2025)."},{"key":"e_1_3_4_30_2","unstructured":"I. D. Raji E. M. Bender A. Paullada E. Denton A. Hanna AI and the everything in the whole wide world benchmark. arXiv [Preprint] (2021). http:\/\/arxiv.org\/abs\/2111.15366 (Accessed 3 September 2025)."},{"key":"e_1_3_4_31_2","unstructured":"L. Orr (2020) Chasing the tail with self-supervised named entity disambiguation. arXiv [Preprint] (2020). http:\/\/arxiv.org\/abs\/2010.10363 (Accessed 3 September 2025)."},{"key":"e_1_3_4_32_2","unstructured":"M. Dahn Our commitment to our customers (2024). https:\/\/www.legalcurrent.com\/our-commitment-to-our-customers\/. Accessed 3 September 2025."},{"key":"e_1_3_4_33_2","first-page":"449","article-title":"Evaluating shepard\u2019s, keycite, and bcite for case validation accuracy","volume":"110","author":"Hellyer P.","year":"2018","unstructured":"P. Hellyer, Evaluating shepard\u2019s, keycite, and bcite for case validation accuracy. Law Libr. J. 110, 449 (2018).","journal-title":"Law Libr. J."},{"key":"e_1_3_4_34_2","doi-asserted-by":"crossref","unstructured":"P. Henderson \u201cDeep reinforcement learning that matters\u201d in Proceedings of the AAAI Conference on Artificial Intelligence (2018). vol. 32.","DOI":"10.1609\/aaai.v32i1.11694"},{"key":"e_1_3_4_35_2","unstructured":"D. Card With little power comes great responsibility. arXiv [Preprint] (2020). http:\/\/arxiv.org\/abs\/2010.06595 (Accessed 3 September 2025)."},{"key":"e_1_3_4_36_2","unstructured":"Y. Ma P. Liang Massive multitask language understanding (MMLU) on helm (2024). https:\/\/crfm.stanford.edu\/2024\/05\/01\/helm-mmlu.html. Accessed 3 September 2025."},{"key":"e_1_3_4_37_2","unstructured":"S. Shankar R. Garcia J. M. Hellerstein A. G. Parameswaran Operationalizing machine learning: An interview study. arXiv [Preprint] (2022). http:\/\/arxiv.org\/abs\/2209.09125 (Accessed 3 September 2025)."},{"key":"e_1_3_4_38_2","doi-asserted-by":"crossref","unstructured":"L. Oakden-Rayner J. Dunnmon G. Carneiro C. R\u00e9 \u201cHidden stratification causes clinically meaningful failures in machine learning for medical imaging\u201d in Proceedings of the ACM Conference on Health Inference and Learning (2020) pp. 151\u2013159.","DOI":"10.1145\/3368555.3384468"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2020"},{"key":"e_1_3_4_40_2","unstructured":"S. Singh The leaderboard illusion. arXiv [Preprint] (2025). http:\/\/arxiv.org\/abs\/2504.20879 (Accessed 3 September 2025)."},{"key":"e_1_3_4_41_2","unstructured":"K. Robison Meta got caught gaming AI benchmarks (2025). https:\/\/www.theverge.com\/meta\/645012\/meta-llama-4-maverick-benchmarks-gaming. Accessed 3 September 2025."},{"key":"e_1_3_4_42_2","doi-asserted-by":"crossref","unstructured":"Y. Wang \u201cMMLU-pro: A more robust and challenging multi-task language understanding benchmark\u201d in The Thirty-Eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024).","DOI":"10.52202\/079017-3018"},{"key":"e_1_3_4_43_2","unstructured":"K. Wiggers Meta exec denies the company artificially boosted llama 4\u2019s benchmark scores (2025). https:\/\/techcrunch.com\/2025\/04\/07\/meta-exec-denies-the-company-artificially-boosted-llama-4s-benchmark-scores\/. Accessed 3 September 2025."},{"key":"e_1_3_4_44_2","unstructured":"T. Simmonite Why and how baidu cheated an artificial intelligence test (2015). https:\/\/www.technologyreview.com\/2015\/06\/04\/72951\/why-and-how-baidu-cheated-an-artificial-intelligence-test\/. Accessed 3 September 2025."},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-1485"},{"key":"e_1_3_4_46_2","doi-asserted-by":"crossref","unstructured":"C. Deng Y. Zhao X. Tang M. Gerstein A. Cohan \u201cInvestigating data contamination in modern benchmarks for large language models\u201d in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) K. Duh H. Gomez S. Bethard Eds. (Association for Computational Linguistics Mexico City Mexico 2024) pp. 8706\u20138719.","DOI":"10.18653\/v1\/2024.naacl-long.482"},{"key":"e_1_3_4_47_2","doi-asserted-by":"crossref","unstructured":"M. C. Elish E. A. Watkins A study of integrating AI in clinical care Repairing innovation (2020).","DOI":"10.69985\/VCVU5761"},{"key":"e_1_3_4_48_2","doi-asserted-by":"crossref","unstructured":"M. M. Mello N. Guha Understanding liability risk from using health care artificial intelligence tools. N. Engl. J. Med. 390 271\u2013278 (2024).","DOI":"10.1056\/NEJMhle2308901"},{"key":"e_1_3_4_49_2","doi-asserted-by":"crossref","unstructured":"D. Schwarcz AI-powered lawyering: AI reasoning models retrieval augmented generation and the future of legal practice (2025).","DOI":"10.2139\/ssrn.5162111"},{"key":"e_1_3_4_50_2","doi-asserted-by":"crossref","unstructured":"A. Nielsen S. Skylaki M. Norkute A. Stremitzer \u201cEffects of xai on legal process\u201d in Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (2023) pp. 442\u2013446.","DOI":"10.1145\/3594536.3595128"},{"key":"e_1_3_4_51_2","first-page":"903","article-title":"Generative AI and legal aid: Results from a field study and 100 use cases to bridge the access to justice gap","volume":"57","author":"Chien C. V.","year":"2024","unstructured":"C. V. Chien, M. Kim, Generative AI and legal aid: Results from a field study and 100 use cases to bridge the access to justice gap. Loy. LAL Rev. 57, 903 (2024).","journal-title":"Loy. LAL Rev."},{"key":"e_1_3_4_52_2","unstructured":"J. Pereyra Introducing biglaw bench (2024). https:\/\/www.harvey.ai\/blog\/introducing-biglaw-bench. Accessed 3 September 2025."},{"key":"e_1_3_4_53_2","first-page":"1","article-title":"Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review","volume":"17","author":"Grossman M. R.","year":"2010","unstructured":"M. R. Grossman, G. V. Cormack, Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Rich. JL Tech. 17, 1 (2010).","journal-title":"Rich. JL Tech."},{"key":"e_1_3_4_54_2","unstructured":"P. Rajpurkar Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv [Preprint] (2017). http:\/\/arxiv.org\/abs\/1711.05225 (Accessed 3 September 2025)."},{"key":"e_1_3_4_55_2","unstructured":"A. K. Zhang A framework for evaluating cybersecurity capabilities and risks of language models. arXiv [Preprint] (2024). http:\/\/arxiv.org\/abs\/2408.08926 (Accessed 3 September 2025)."},{"key":"e_1_3_4_56_2","unstructured":"J. Vanschoren S. Yeung Announcing the neurips 2021 datasets and benchmarks track (2021). https:\/\/neuripsconf.medium.com\/announcing-the-neurips-2021-datasets-and-benchmarks-track-644e27c1e66c. Accessed 3 September 2025."},{"key":"e_1_3_4_57_2","unstructured":"F. Chollet M. Knoop G. Kamradt B. Landers Arc prize 2024: Technical report. arXiv [Preprint] (2024). http:\/\/arxiv.org\/abs\/2412.04604 (Accessed 3 September 2025)."},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"e_1_3_4_59_2","unstructured":"R. K. Arora Evaluating large language models towards improved human health. arXiv [Preprint] (2025). http:\/\/arxiv.org\/abs\/2505.08775 (Accessed 3 September 2025)."},{"key":"e_1_3_4_60_2","unstructured":"M. Chen Evaluating large language models trained on code. arXiv [Preprint] (2021). http:\/\/arxiv.org\/abs\/2107.03374 (Accessed 3 September 2025)."},{"key":"e_1_3_4_61_2","first-page":"387","article-title":"The algorithm as a human artifact: Implications for legal [re] search","volume":"109","author":"Mart S. N.","year":"2017","unstructured":"S. N. Mart, The algorithm as a human artifact: Implications for legal [re] search. Law Libr. J. 109, 387 (2017).","journal-title":"Law Libr. J."},{"key":"e_1_3_4_62_2","doi-asserted-by":"crossref","unstructured":"M. Ma B. Waldon J. Nyarko \u201cConceptual questions in developing expert-annotated data\u201d in Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (2023) pp. 427\u2013431.","DOI":"10.1145\/3594536.3595139"},{"key":"e_1_3_4_63_2","doi-asserted-by":"crossref","unstructured":"M. L. Gordon K. Zhou K. Patel T. Hashimoto M. S. Bernstein \u201cThe disagreement deconvolution: Bringing machine learning performance metrics in line with reality\u201d in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021) pp. 1\u201314.","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_3_4_64_2","unstructured":"J. Forward Empirics in law: A scientific approach to improving access to justice (2019). https:\/\/www.wisbar.org\/newspublications\/insidetrack\/pages\/Article.aspx?Volume=11&Issue=2&ArticleID=26821. Accessed 3 September 2025."},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1162\/daed_a_00536"},{"key":"e_1_3_4_66_2","doi-asserted-by":"crossref","unstructured":"D. L. Sackett W. M. Rosenberg J. M. Gray R. B. Haynes W. S. Richardson Evidence based medicine: what it is and what it isn\u2019t. BMJ. 312 71\u201372 (1996).","DOI":"10.1136\/bmj.312.7023.71"},{"key":"e_1_3_4_67_2","unstructured":"A. B. Association Rule 1.6 confidentiality of information - comment (2025). https:\/\/www.americanbar.org\/groups\/professional_responsibility\/publications\/model_rules_of_professional_conduct\/rule_1_6_confidentiality_of_information\/comment_on_rule_1_6\/. Accessed 3 September 2025."},{"key":"e_1_3_4_68_2","doi-asserted-by":"crossref","unstructured":"S. H. Cen R. Alur \u201cFrom transparency to accountability and back: A discussion of access and evidence in AI auditing\u201d in Proceedings of the 4th ACM Conference on Equity and Access in Algorithms Mechanisms and Optimization EAAMO\u201924 (Association for Computing Machinery New York NY USA 2024).","DOI":"10.1145\/3689904.3694711"},{"key":"e_1_3_4_69_2","first-page":"893","article-title":"The necessary and proper stewardship of judicial data","volume":"76","author":"Clopton Z. D.","year":"2024","unstructured":"Z. D. Clopton, A. Z. Huq, The necessary and proper stewardship of judicial data. Stan. L. Rev. 76, 893 (2024).","journal-title":"Stan. L. Rev."},{"key":"e_1_3_4_70_2","first-page":"27","article-title":"Full-text databases and legal research: Backing into the future","volume":"1","author":"Berring R. C.","year":"1986","unstructured":"R. C. Berring, Full-text databases and legal research: Backing into the future. High Tech. LJ 1, 27 (1986).","journal-title":"High Tech. LJ"},{"key":"e_1_3_4_71_2","first-page":"797","article-title":"Open access in a closed universe: Lexis, westlaw, law schools, and the legal information market","volume":"10","author":"Arewa O. B.","year":"2006","unstructured":"O. B. Arewa, Open access in a closed universe: Lexis, westlaw, law schools, and the legal information market. Lewis Clark L. Rev. 10, 797 (2006).","journal-title":"Lewis Clark L. Rev."},{"key":"e_1_3_4_72_2","unstructured":"S. Biderman \u201cPythia: A suite for analyzing large language models across training and scaling\u201d in International Conference on Machine Learning (PMLR 2023) pp. 2397\u20132430."},{"key":"e_1_3_4_73_2","unstructured":"ABA Aba issues first ethics guidance on a lawyer\u2019s use of AI tools (2024). https:\/\/www.americanbar.org\/news\/abanews\/aba-news-archives\/2024\/07\/aba-issues-first-ethics-guidance-ai-tools\/. Accessed 3 September 2025."},{"key":"e_1_3_4_74_2","unstructured":"V. Beiser Ai & the law. & what it means for legal education & lawyers (2024). https:\/\/www.law.georgetown.edu\/news\/ai-the-law-what-it-means-for-legal-education-lawyers\/. Accessed 3 September 2025."},{"key":"e_1_3_4_75_2","first-page":"69","article-title":"Is disclosure and certification of the use of generative AI really necessary?","volume":"107","author":"Grimm P. W.","year":"2023","unstructured":"P. W. Grimm, M. R. Grossman, D. G. Brown, Is disclosure and certification of the use of generative AI really necessary? Judicature 107, 69\u201377 (2023).","journal-title":"Judicature"},{"key":"e_1_3_4_76_2","unstructured":"NIST Face recognition vendor test (frvt) 2002 (2022). https:\/\/www.nist.gov\/itl\/iad\/image-group\/face-recognition-vendor-test-frvt-2002. Accessed 3 September 2025."},{"key":"e_1_3_4_77_2","unstructured":"V. J. Reddi C. J. Wu D. Kanter P. Mattson \u201cRetrospective: Mlperf\u201d (Tech. Rep. MLPerf 2023) https:\/\/bpb-us-w2.wpmucdn.com\/sites.coecis.cornell.edu\/dist\/7\/587\/files\/2023\/06\/JanapaReddi_2020_MLPerf.pdf."},{"key":"e_1_3_4_78_2","unstructured":"K. Roose How claude became tech insiders\u2019 chatbot of choice (2024). https:\/\/www.nytimes.com\/2024\/12\/13\/technology\/claude-ai-anthropic.html. Accessed 3 September 2025."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2509757122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T19:17:24Z","timestamp":1784575044000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2509757122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,20]]},"references-count":78,"journal-issue":{"issue":"30","published-print":{"date-parts":[[2026,7,28]]}},"alternative-id":["10.1073\/pnas.2509757122"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2509757122","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,20]]},"assertion":[{"value":"2026-07-20","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2509757122"}}