{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T23:30:32Z","timestamp":1774049432779,"version":"3.50.1"},"reference-count":23,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T00:00:00Z","timestamp":1719273600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"name":"European Union's Horizon Research and Innovation","award":["101057062"],"award-info":[{"award-number":["101057062"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To assess the performance of large language models (LLMs) for zero-shot disambiguation of acronyms in clinical narratives.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Clinical narratives in English, German, and Portuguese were applied for testing the performance of four LLMs: GPT-3.5, GPT-4, Llama-2-7b-chat, and Llama-2-70b-chat. For English, the anonymized Clinical Abbreviation Sense Inventory (CASI, University of Minnesota) was used. For German and Portuguese, at least 500 text spans were processed. The output of LLM models, prompted with contextual information, was analyzed to compare their acronym disambiguation capability, grouped by document-level metadata, the source language, and the LLM.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>On CASI, GPT-3.5 achieved 0.91 in accuracy. GPT-4 outperformed GPT-3.5 across all datasets, reaching 0.98 in accuracy for CASI, 0.86 and 0.65 for two German datasets, and 0.88 for Portuguese. Llama models only reached 0.73 for CASI and failed severely for German and Portuguese. Across LLMs, performance decreased from English to German and Portuguese processing languages. There was no evidence that additional document-level metadata had a significant effect.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>For English clinical narratives, acronym resolution by GPT-4 can be recommended to improve readability of clinical text by patients and professionals. For German and Portuguese, better models are needed. Llama models, which are particularly interesting for processing sensitive content on premise, cannot yet be recommended for acronym resolution.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae157","type":"journal-article","created":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T21:15:34Z","timestamp":1719350134000},"page":"2040-2046","source":"Crossref","is-referenced-by-count":10,"title":["Disambiguation of acronyms in clinical narratives with large language models"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4674-0146","authenticated-orcid":false,"given":"Amila","family":"Kugic","sequence":"first","affiliation":[{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz , 8036 Graz, Austria"}]},{"given":"Stefan","family":"Schulz","sequence":"additional","affiliation":[{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz , 8036 Graz, Austria"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9824-9004","authenticated-orcid":false,"given":"Markus","family":"Kreuzthaler","sequence":"additional","affiliation":[{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz , 8036 Graz, Austria"}]}],"member":"286","published-online":{"date-parts":[[2024,6,25]]},"reference":[{"issue":"6","key":"2024082207515746100_ocae157-B1","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1111\/jep.13533","article-title":"Structure, content, unsafe abbreviations, and completeness of discharge summaries: a retrospective analysis in a University Hospital in Austria","volume":"27","author":"Schwarz","year":"2021","journal-title":"J Eval Clin Pract"},{"key":"2024082207515746100_ocae157-B2","doi-asserted-by":"crossref","first-page":"104101","DOI":"10.1016\/j.ijmedinf.2020.104101","article-title":"The CLASSE GATOR (CLinical Acronym SenSE disambiGuATOR): a Method for predicting acronym sense from neonatal clinical notes","volume":"137","author":"Kashyap","year":"2020","journal-title":"Int J Med Inform"},{"issue":"1","key":"2024082207515746100_ocae157-B3","doi-asserted-by":"crossref","first-page":"5319","DOI":"10.1038\/s41467-021-25578-4","article-title":"Automatically disambiguating medical acronyms with ontology-aware deep learning","volume":"12","author":"Skreta","year":"2021","journal-title":"Nat Commun"},{"key":"2024082207515746100_ocae157-B4","author":"Moon","year":"2012"},{"key":"2024082207515746100_ocae157-B5","first-page":"12","article-title":"Zero-Shot clinical acronym expansion via latent meaning cells","volume":"136","author":"Adams","year":"2020","journal-title":"Proc Mach Learn Res"},{"key":"2024082207515746100_ocae157-B6","author":"Vaswani"},{"key":"2024082207515746100_ocae157-B7","doi-asserted-by":"crossref","first-page":"1169595","DOI":"10.3389\/frai.2023.1169595","article-title":"ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations","volume":"6","author":"Dave","year":"2023","journal-title":"Front Artif Intell"},{"issue":"12","key":"2024082207515746100_ocae157-B8","doi-asserted-by":"crossref","first-page":"2647","DOI":"10.1007\/s10439-023-03284-0","article-title":"ChatGPT, Bard, and large language models for biomedical research: opportunities and pitfalls","volume":"51","author":"Thapa","year":"2023","journal-title":"Ann Biomed Eng"},{"issue":"3","key":"2024082207515746100_ocae157-B9","doi-asserted-by":"crossref","first-page":"e107","DOI":"10.1016\/S2589-7500(23)00021-3","article-title":"ChatGPT: the future of discharge summaries?","volume":"5","author":"Patel","year":"2023","journal-title":"Lancet Digital Health"},{"issue":"3","key":"2024082207515746100_ocae157-B10","first-page":"123","article-title":"ChatGPT\u2019s ability to assist with clinical documentation: a randomized controlled trial","volume":"32","author":"Baker","year":"2023","journal-title":"J Am Acad Orthop Surg"},{"key":"2024082207515746100_ocae157-B11","doi-asserted-by":"crossref","first-page":"385","DOI":"10.18653\/v1\/2023.clinicalnlp-1.41","volume-title":"Proceedings of the 5th Clinical Natural Language Processing Workshop","author":"Ramachandran","year":"2023"},{"issue":"8","key":"2024082207515746100_ocae157-B12","doi-asserted-by":"crossref","first-page":"1367","DOI":"10.1093\/jamia\/ocad012","article-title":"The 2022 n2c2\/UW shared task on extracting social determinants of health","volume":"30","author":"Lybarger","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2024082207515746100_ocae157-B13","doi-asserted-by":"crossref","first-page":"503","DOI":"10.18653\/v1\/2023.clinicalnlp-1.52","volume-title":"Proceedings of the 5th Clinical Natural Language Processing Workshop","author":"Ben Abacha","year":"2023"},{"key":"2024082207515746100_ocae157-B14","first-page":"78","article-title":"Clinical acronym disambiguation via ChatGPT and BING","volume":"309","author":"Kugic","year":"2023","journal-title":"Stud Health Technol Inform"},{"key":"2024082207515746100_ocae157-B15","doi-asserted-by":"crossref","first-page":"104753","DOI":"10.1016\/j.ijmedinf.2022.104753","article-title":"Binary acronym disambiguation in clinical notes from electronic health records with an application in computational phenotyping","volume":"162","author":"Link","year":"2022","journal-title":"Int J Med Inform"},{"key":"2024082207515746100_ocae157-B16","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1016\/j.jclinepi.2015.09.016","article-title":"Million Veteran Program: a mega-biobank to study genetic influences on health and disease","volume":"70","author":"Gaziano","year":"2016","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2024082207515746100_ocae157-B17","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1186\/s13326-022-00269-1","article-title":"SemClinBr\u2014a multi-institutional and multi-specialty semantically annotated corpus for Portuguese clinical NLP tasks","volume":"13","author":"Oliveira","year":"2022","journal-title":"J Biomed Semant"},{"key":"2024082207515746100_ocae157-B18","doi-asserted-by":"crossref","first-page":"1998","DOI":"10.18653\/v1\/2022.emnlp-main.130","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Agrawal","year":"2022"},{"issue":"2","key":"2024082207515746100_ocae157-B19","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s00270-023-03563-2","article-title":"Feasibility of GPT-3 and GPT-4 for in-depth patient education prior to interventional radiological procedures: a comparative analysis","volume":"47","author":"Scheschenja","year":"2023","journal-title":"Cardiovasc Intervent Radiol"},{"issue":"1","key":"2024082207515746100_ocae157-B20","doi-asserted-by":"crossref","first-page":"18562","DOI":"10.1038\/s41598-023-45837-2","article-title":"Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology","volume":"13","author":"Taloni","year":"2023","journal-title":"Sci Rep"},{"key":"2024082207515746100_ocae157-B21","first-page":"738","author":"Dreano","year":"2023"},{"issue":"5","key":"2024082207515746100_ocae157-B22","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1111\/imj.16393","article-title":"The new paradigm in machine learning\u2014foundation models, large language models and beyond: a primer for physicians","volume":"54","author":"Scott","year":"2024","journal-title":"Intern Med J"},{"issue":"1","key":"2024082207515746100_ocae157-B23","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1038\/s41746-023-00879-8","article-title":"The shaky foundations of large language models and foundation models for electronic health records","volume":"6","author":"Wornow","year":"2023","journal-title":"NPJ Digit Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2040\/58868208\/ocae157.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2040\/58868208\/ocae157.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T11:52:58Z","timestamp":1724327578000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/2040\/7699035"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,25]]},"references-count":23,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,6,25]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae157","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,6,25]]}}}