{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T21:54:25Z","timestamp":1775685265299,"version":"3.50.1"},"reference-count":65,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T00:00:00Z","timestamp":1700179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>The emergence of ChatGPT has sensitized the general public, including the legal profession, to large language models' (LLMs) potential uses (e.g., document drafting, question answering, and summarization). Although recent studies have shown how well the technology performs in diverse semantic annotation tasks focused on legal texts, an influx of newer, more capable (GPT-4) or cost-effective (GPT-3.5-turbo) models requires another analysis. This paper addresses recent developments in the ability of LLMs to semantically annotate legal texts in zero-shot learning settings. Given the transition to mature generative AI systems, we examine the performance of GPT-4 and GPT-3.5-turbo(-16k), comparing it to the previous generation of GPT models, on three legal text annotation tasks involving diverse documents such as adjudicatory opinions, contractual clauses, or statutory provisions. We also compare the models' performance and cost to better understand the trade-offs. We found that the GPT-4 model clearly outperforms the GPT-3.5 models on two of the three tasks. The cost-effective GPT-3.5-turbo matches the performance of the 20\u00d7 more expensive text-davinci-003 model. While one can annotate multiple data points within a single prompt, the performance degrades as the size of the batch increases. This work provides valuable information relevant for many practical applications (e.g., in contract review) and research projects (e.g., in empirical legal studies). Legal scholars and practicing lawyers alike can leverage these findings to guide their decisions in integrating LLMs in a wide range of workflows involving semantic annotation of legal texts.<\/jats:p>","DOI":"10.3389\/frai.2023.1279794","type":"journal-article","created":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T11:15:00Z","timestamp":1700219700000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":54,"title":["The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts"],"prefix":"10.3389","volume":"6","author":[{"given":"Jaromir","family":"Savelka","sequence":"first","affiliation":[]},{"given":"Kevin D.","family":"Ashley","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,11,17]]},"reference":[{"key":"B1","first-page":"3","article-title":"\u201cIdentification of rhetorical roles of sentences in Indian legal judgments,\u201d","volume-title":"JURIX 2019, Vol. 322","author":"Bhattacharya","year":"2019"},{"key":"B2","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1145\/1165485.1165506","article-title":"\u201cAutomatic semantics extraction in law documents,\u201d","volume-title":"Proceedings of the 10th International Conference on Artificial Intelligence and Law","author":"Biagioli","year":"2005"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1145\/3594536.3595163","article-title":"Can gpt-3 perform statutory reasoning?","author":"Blair-Stanek","year":"2023","journal-title":"arXiv"},{"key":"B4","first-page":"21","article-title":"\u201cMulti-label classification of legislative text into eurovoc,\u201d","volume-title":"Legal Knowledge and Information Systems: JURIX 2012: the Twenty-fifth Annual Conference, Vol. 250","author":"Boella","year":"2012"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4322372","article-title":"Gpt as knowledge worker: a zero-shot evaluation of (ai)cpa capabilities","author":"Bommarito","year":"2023","journal-title":"arXiv"},{"key":"B6","article-title":"\u201cPerformance in the courtroom: automated processing and visualization of appeal court decisions in France,\u201d","author":"Boniol","year":"2020","journal-title":"Proceedings of the Natural Legal Language Processing Workshop"},{"key":"B7","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1145\/3322640.3326723","article-title":"\u201cSemi-supervised methods for explainable legal prediction,\u201d","volume-title":"ICAIL","author":"Branting","year":"2019"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.5555\/3495724.3495883","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B9","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1145\/3086512.3086515","article-title":"\u201cExtracting contract elements,\u201d","volume-title":"Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law","author":"Chalkidis","year":"2017"},{"key":"B10","first-page":"2898","article-title":"\u201cLegal-bert: the muppets straight out of law school,\u201d","author":"Chalkidis","year":"2020","journal-title":"Findings of the Association for Computational Linguistics: EMNLP"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2101.04355","article-title":"Neural contract element extraction revisited: letters from sesame street","author":"Chalkidis","year":"2021","journal-title":"arXiv"},{"key":"B12","author":"Chen","year":"2021","journal-title":"Evaluating Large Language Models Trained on Code"},{"key":"B13","first-page":"87","article-title":"\u201cMachine learning versus knowledge based classification of legal texts,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"de Maat","year":"2010"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv"},{"key":"B15","article-title":"\u201cLetsum, an automatic text summarization system in law field,\u201d","volume-title":"JURIX","author":"Farzindar","year":"2004"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-12837-0_6","volume-title":"Integrating a Bottom-Up and Top-Down Methodology for Building Semantic Resources for the Multilingual Legal Domain","author":"Francesconi","year":"2010"},{"key":"B17","first-page":"101","article-title":"\u201cAutomatic identification and empirical analysis of legally relevant factors,\u201d","volume-title":"ICAIL","author":"Gray","year":"2023"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.05327","article-title":"Blind judgement: agent-based supreme court modelling with gpt","author":"Hamilton","year":"2023","journal-title":"arXiv"},{"key":"B19","volume-title":"Automatic Segmentation of Czech Court Decisions Into Multi-paragraph Parts","author":"Harasta","year":"2019"},{"key":"B20","article-title":"\u201cCuad: an expert-annotated nlp dataset for legal contract review,\u201d","author":"Hendrycks","year":"2021","journal-title":"35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks"},{"key":"B21","first-page":"278","article-title":"\u201cRandom decision forests,\u201d","volume-title":"Proceedings of 3rd International Conference on Document Analysis and Recognition, Vol. 1","author":"Ho","year":"1995"},{"key":"B22","author":"Katz","year":"2023","journal-title":"Gpt-4 Passes the Bar Exam"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1412.6980","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.11942","article-title":"Albert: a lite bert for self-supervised learning of language representations","author":"Lan","year":"2019","journal-title":"arXiv"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2010.10386","article-title":"A benchmark for lease contract review","author":"Leivaditi","year":"2020","journal-title":"arXiv"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692","article-title":"Roberta: a robustly optimized bert pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.06912","article-title":"How well do sota legal reasoning models support abductive reasoning?","author":"Nguyen","year":"2023","journal-title":"arXiv"},{"key":"B28","year":"2023","journal-title":"Gpt-4 Technical Report"},{"key":"B29","article-title":"\u201cTraining language models to follow instructions with human feedback,\u201d","volume-title":"Advances in Neural Information Processing Systems","author":"Ouyang","year":"2022"},{"key":"B30","author":"Perlman","year":"2022","journal-title":"The Implications of Openai's Assistant for Legal Services and Society"},{"key":"B31","first-page":"133","article-title":"\u201cExtracting outcomes from appellate decisions in US state courts,\u201d","volume-title":"JURIX","author":"Petrova","year":"2020"},{"key":"B32","first-page":"67","article-title":"\u201cEchr: legal corpus for argument mining,\u201d","volume-title":"Proceedings of the 7th Workshop on Argument Mining","author":"Poudyal","year":"2020"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.cs\/0609059","article-title":"Automatic annotation of multilingual text collections with a conceptual thesaurus","author":"Pouliquen","year":"2006","journal-title":"arXiv"},{"key":"B34","author":"Radford","year":"2018","journal-title":"Improving Language Understanding by Generative Pre-training"},{"key":"B35","author":"Radford","year":"2019","journal-title":"Language Models are Unsupervised Multitask Learners"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019","journal-title":"arXiv"},{"key":"B37","doi-asserted-by":"crossref","first-page":"102","DOI":"10.18653\/v1\/2021.nllp-1.10","article-title":"\u201cFew-shot and zero-shot approaches to legal text classification: a case study in the financial sector,\u201d","volume-title":"Proceedings of the Natural Legal Language Processing Workshop 2021","author":"Sarkar","year":"2021"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1145\/3594536.3595161","article-title":"Unlocking practical applications in legal domain: evaluation of gpt for zero-shot semantic annotation of legal texts","author":"Savelka","year":"2023","journal-title":"arXiv"},{"key":"B39","article-title":"\u201cUsing conditional random fields to detect different functional types of content in decisions of United States courts with example application to sentence boundary detection,\u201d","author":"Savelka","year":"2017","journal-title":"Workshop on Automated Semantic Analysis of Information in Legal Texts"},{"key":"B40","first-page":"111","article-title":"\u201cSegmenting us court decisions into functional and issue specific parts,\u201d","volume-title":"JURIX","author":"Savelka","year":"2018"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10506-021-09293-5","article-title":"Legal information retrieval for understanding statutory terms","volume":"30","author":"Savelka","year":"2022","journal-title":"Artif. Intell. Law"},{"key":"B42","unstructured":"Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise?112\n            SavelkaJ.\n            AshleyK. D.\n            GrayM.\n            WestermannH.\n            XuH.\n          Automat. Semant. Anal. Inf. Legal Text34412023"},{"key":"B43","first-page":"133","article-title":"\u201cMining information from statutory texts in multi-jurisdictional settings,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"Savelka","year":"2014"},{"key":"B44","article-title":"Sentence boundary detection in adjudicatory decisions in the united states","author":"Savelka","year":"2017","journal-title":"Traitement Automat. Lang"},{"key":"B45","author":"Savelka","year":"2020","journal-title":"Cross-Domain Generalization and Knowledge Transfer in Transformers Trained on Legal Data"},{"key":"B46","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1145\/3462757.3466149","article-title":"\u201cLex rosetta: transfer of predictive models across languages, jurisdictions, and legal domains,\u201d","volume-title":"Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law","author":"Savelka","year":"2021"},{"key":"B47","first-page":"53","article-title":"\u201cNetwork analysis of manually-encoded state laws and prospects for automatio,\u201d","author":"Sweeney","year":"2013","journal-title":"Network Analysis in Law"},{"key":"B48","article-title":"\u201cChatgpt as an artificial lawyer?,\u201d","volume-title":"Artificial Intelligence for Access to Justice (AI4AJ 2023)","author":"Tan","year":"2023"},{"key":"B49","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B50","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1145\/3086512.3086535","article-title":"\u201cSemantic types for computational legal reasoning: propositional connectives and sentence roles in the veterans' claims dataset,\u201d","volume-title":"Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law","author":"Walker","year":"2017"},{"key":"B51","article-title":"\u201cAutomatic classification of rhetorical roles for sentences: comparing rule-based scripts with machine learning,\u201d","volume-title":"ASAIL@ ICAIL 2385","author":"Walker","year":"2019"},{"key":"B52","unstructured":"Maud: an expert-annotated legal nlp dataset for merger agreement understanding\n            WangS. H.\n            ScardigliA.\n            TangL.\n            ChenW.\n            LevkinD.\n            ChenA.\n          arXiv [preprint]2023"},{"key":"B53","article-title":"\u201cLlmediator: Gpt-4 assisted online dispute resolution,\u201d","volume-title":"Artificial Intelligence for Access to Justice (AI4AJ 2023","author":"Westermann","year":"2023"},{"key":"B54","first-page":"123","article-title":"\u201cComputer-assisted creation of boolean search rules for text classification in the legal domain,\u201d","volume-title":"JURIX, Vol. 322","author":"Westermann","year":"2019"},{"key":"B55","first-page":"164","article-title":"\u201cSentence embeddings and high-speed similarity search for fast computer assisted annotation of legal documents,\u201d","volume-title":"JURIX, Vol. 334","author":"Westermann","year":"2020"},{"key":"B56","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3233\/FAIA210316","article-title":"\u201cData-centric machine learning: Improving model performance and understanding through dataset analysis,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"Westermann","year":"2021"},{"key":"B57","first-page":"157","article-title":"\u201cAutomatic extraction of legal concepts and definitions,\u201d","volume-title":"Legal Knowledge and Information Systems: JURIX 2012: the Twenty-Fifth Annual Conference, Vol. 250","author":"Winkels","year":"2012"},{"key":"B58","first-page":"113","article-title":"\u201cOn rule extraction from regulations,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"Wyner","year":"2011"},{"key":"B59","unstructured":"Argumentative segmentation enhancement for legal summarization141150\n            XuH.\n            AshleyK. D.\n          Automat. Semant. Anal. Inf. Legal Text34412023"},{"key":"B60","doi-asserted-by":"crossref","DOI":"10.3233\/FAIA200862","article-title":"\u201cUsing argument mining for legal text summarization,\u201d","volume-title":"JURIX, Vol","author":"Xu","year":"2020"},{"key":"B61","first-page":"33","article-title":"\u201cAccounting for sentence position and legal domain sentence embedding in learning to classify case sentences,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"Xu","year":""},{"key":"B62","first-page":"250","article-title":"\u201cToward summarizing case decisions via extracting argument issues, reasons, and conclusions,\u201d","volume-title":"Proceedings of the 18th International Conference on Artificial Intelligence and Law","author":"Xu","year":""},{"key":"B63","author":"Yu","year":"2022","journal-title":"Legal Prompting: Teaching a Language Model to Think Like a Lawyer"},{"key":"B64","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1145\/3462757.3466088","article-title":"\u201cWhen does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings,\u201d","volume-title":"Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law","author":"Zheng","year":"2021"},{"key":"B65","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1145\/3322640.3326728","article-title":"\u201cAutomatic summarization of legal decisions using iterative masking of predictive sentences,\u201d","volume-title":"ICAIL","author":"Zhong","year":"2019"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1279794\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T11:15:32Z","timestamp":1700219732000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2023.1279794\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,17]]},"references-count":65,"alternative-id":["10.3389\/frai.2023.1279794"],"URL":"https:\/\/doi.org\/10.3389\/frai.2023.1279794","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,17]]},"article-number":"1279794"}}