{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T08:40:07Z","timestamp":1780389607056,"version":"3.54.1"},"posted":{"date-parts":[[2025,5,25]]},"group-title":"Health Informatics","reference-count":49,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,5,25]],"date-time":"2025-05-25T00:00:00Z","timestamp":1748131200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,9,26]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                <jats:sec>\n                  <jats:title>Context<\/jats:title>\n                  <jats:p>Goals-of-care (GOC) discussions and their documentation are important process measures in palliative care. However, existing natural language processing (NLP) models for identifying such documentation require costly task-specific training data. Large language models (LLMs) hold promise for measuring such constructs with fewer or no task-specific training data.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To evaluate the performance of a publicly available LLM with no task-specific training data (zero-shot prompting) for identifying documented GOC discussions.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>\n                    We compared performance of two NLP models in identifying documented GOC discussions: Llama 3.3 using zero-shot prompting; and, a task-specific BERT (Bidirectional Encoder Representations from Transformers)-based model trained on 4,642 manually annotated notes. We tested both models on records from a series of clinical trials enrolling adult patients with chronic life-limiting illness hospitalized over 2018-2023. We evaluated the area under the receiver operating characteristic curve (AUC), area under the precision-recall curve (AUPRC), and maximal F\n                    <jats:sub>1<\/jats:sub>\n                    score, for both note-level and patient-level classification over a 30-day period.\n                  <\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>\n                    In our text corpora, GOC documentation represented &lt;1% of text and was found in 7.3-9.9% of notes for 23-37% of patients. In a 617-patient held-out test set, Llama 3.3 (zero-shot) and BERT (task-specific, trained) exhibited comparable performance in identifying GOC documentation (Llama 3.3: AUC 0.979, AUPRC 0.873, and F\n                    <jats:sub>1<\/jats:sub>\n                    0.83; BERT: AUC 0.981, AUPRC 0.874, and F\n                    <jats:sub>1<\/jats:sub>\n                    0.83).\n                  <\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>A zero-shot large language model with no task-specific training performed similarly to a task-specific trained BERT model in identifying documented goals-of-care discussions. This demonstrates the promise of LLMs in measuring novel clinical research outcomes.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>KEY MESSAGE<\/jats:title>\n                  <jats:p>This article reports the performance of a publicly available large language model with no task-specific training data in measuring the occurrence of documented goals-of-care discussions from electronic health records. The study demonstrates that newer large language AI models may allow investigators to measure novel outcomes without requiring costly training data.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2025.05.23.25328115","type":"posted-content","created":{"date-parts":[[2025,5,25]],"date-time":"2025-05-25T12:30:11Z","timestamp":1748176211000},"source":"Crossref","is-referenced-by-count":2,"title":["Assessment of a zero-shot large language model in measuring documented goals-of-care discussions"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5535-287X","authenticated-orcid":false,"given":"Robert Y.","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin S.","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Sibley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Trevor","family":"Cohen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William B.","family":"Lober","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danae G.","family":"Dotolo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erin K.","family":"Kross","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2025092903550797000_2025.05.23.25328115v2.1","doi-asserted-by":"publisher","DOI":"10.1016\/j.nlp.2025.100129"},{"key":"2025092903550797000_2025.05.23.25328115v2.2","doi-asserted-by":"publisher","DOI":"10.2196\/17984"},{"key":"2025092903550797000_2025.05.23.25328115v2.3","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-024-00583-7"},{"key":"2025092903550797000_2025.05.23.25328115v2.4","doi-asserted-by":"publisher","DOI":"10.1126\/science.adq9356"},{"key":"2025092903550797000_2025.05.23.25328115v2.5","unstructured":"Dave Bergmann. What is zero-shot learning? Explainers: Become an expert on emerging and fundamental tech topics. January 24, 2024. Accessed September 10, 2025. https:\/\/www.ibm.com\/think\/topics\/zero-shot-learning"},{"key":"2025092903550797000_2025.05.23.25328115v2.6","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2014.5271"},{"key":"2025092903550797000_2025.05.23.25328115v2.7","doi-asserted-by":"publisher","DOI":"10.1097\/00003246-200102001-00006"},{"key":"2025092903550797000_2025.05.23.25328115v2.8","doi-asserted-by":"publisher","DOI":"10.1513\/AnnalsATS.201612-965OI"},{"key":"2025092903550797000_2025.05.23.25328115v2.9","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2013.180"},{"key":"2025092903550797000_2025.05.23.25328115v2.10","doi-asserted-by":"publisher","DOI":"10.1136\/postgradmedj-2016-133951"},{"key":"2025092903550797000_2025.05.23.25328115v2.11","doi-asserted-by":"publisher","DOI":"10.1097\/CCM.0000000000003711"},{"key":"2025092903550797000_2025.05.23.25328115v2.12","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2024.0496"},{"key":"2025092903550797000_2025.05.23.25328115v2.13","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2012.0472"},{"key":"2025092903550797000_2025.05.23.25328115v2.14","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2014.12.007"},{"key":"2025092903550797000_2025.05.23.25328115v2.15","doi-asserted-by":"crossref","unstructured":"Chien I , Shi A , Chan A , Lindvall C. Identification of Serious Illness Conversations in Unstructured Clinical Notes Using Deep Neural Networks. In: Fernando Koch , Andrew Koster , Isabelle Bichindaritz , Pau Herrero , et al , eds. Artificial Intelligence in Health: First International Workshop, AIH 2018, Stockholm, Sweden, July 13-14, 2018, Revised Selected Papers. Lecture Notes in Artificial Intelligence. Springer Nature Switzerland; 2019:199\u2013212.","DOI":"10.1007\/978-3-030-12738-1_15"},{"key":"2025092903550797000_2025.05.23.25328115v2.16","doi-asserted-by":"publisher","DOI":"10.1177\/0269216318810421"},{"key":"2025092903550797000_2025.05.23.25328115v2.17","doi-asserted-by":"publisher","DOI":"10.1016\/J.JPAINSYMMAN.2020.08.024"},{"key":"2025092903550797000_2025.05.23.25328115v2.18","doi-asserted-by":"publisher","DOI":"10.1200\/OP-24-00992"},{"key":"2025092903550797000_2025.05.23.25328115v2.19","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.1204"},{"key":"2025092903550797000_2025.05.23.25328115v2.20","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.06.025"},{"key":"2025092903550797000_2025.05.23.25328115v2.21","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.8812"},{"key":"2025092903550797000_2025.05.23.25328115v2.22","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.9150"},{"key":"2025092903550797000_2025.05.23.25328115v2.23","doi-asserted-by":"publisher","DOI":"10.1056\/CAT.22.0271"},{"key":"2025092903550797000_2025.05.23.25328115v2.24","doi-asserted-by":"publisher","DOI":"10.12788\/jhm.3714"},{"key":"2025092903550797000_2025.05.23.25328115v2.25","unstructured":"Devlin J , Chang MW , Lee K , Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. Preprint posted online May 24, 2019. Accessed August 29, 2024. http:\/\/arxiv.org\/abs\/1810.04805"},{"key":"2025092903550797000_2025.05.23.25328115v2.26","unstructured":"Alsentzer E , Murphy JR , Boag W , et al. Publicly Available Clinical BERT Embeddings. arXiv. Preprint posted online June 20, 2019. Accessed August 29, 2024. https:\/\/arxiv.org\/abs\/1904.03323"},{"key":"2025092903550797000_2025.05.23.25328115v2.27","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2024.11.016"},{"key":"2025092903550797000_2025.05.23.25328115v2.28","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.5088"},{"key":"2025092903550797000_2025.05.23.25328115v2.29","doi-asserted-by":"publisher","DOI":"10.1016\/j.cct.2022.106879"},{"key":"2025092903550797000_2025.05.23.25328115v2.30","unstructured":"Jack Bandy , Nicholas Vincent . Addressing \u201cDocumentation Debt\u201d in Machine Learning: A Retrospective Datasheet for BookCorpus. In: J. Vanschoren , S. Yeung , eds. 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2021."},{"key":"2025092903550797000_2025.05.23.25328115v2.31","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35"},{"key":"2025092903550797000_2025.05.23.25328115v2.32","unstructured":"Alsentzer E. BioClinicalBERT. 2019. https:\/\/huggingface.co\/emilyalsentzer\/Bio_ClinicalBERT"},{"key":"2025092903550797000_2025.05.23.25328115v2.33","unstructured":"Jacob Devlin , Ming-Wei Chang . Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing. 2018. https:\/\/research.google\/blog\/open-sourcing-bert-state-of-the-art-pre-training-for-natural-language-processing\/"},{"key":"2025092903550797000_2025.05.23.25328115v2.34","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-019-05446-0"},{"key":"2025092903550797000_2025.05.23.25328115v2.35","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2022.02.006"},{"key":"2025092903550797000_2025.05.23.25328115v2.36","doi-asserted-by":"publisher","DOI":"10.1177\/1536867x20909693"},{"key":"2025092903550797000_2025.05.23.25328115v2.37","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2019.0059"},{"key":"2025092903550797000_2025.05.23.25328115v2.38","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.12.020"},{"key":"2025092903550797000_2025.05.23.25328115v2.39","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooae060"},{"key":"2025092903550797000_2025.05.23.25328115v2.40","doi-asserted-by":"publisher","DOI":"10.1111\/head.14702"},{"key":"2025092903550797000_2025.05.23.25328115v2.41","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00970-0"},{"key":"2025092903550797000_2025.05.23.25328115v2.42","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocae260"},{"key":"2025092903550797000_2025.05.23.25328115v2.43","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00704"},{"key":"2025092903550797000_2025.05.23.25328115v2.44","unstructured":"Wei J , Wang X , Schuurmans D , et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In: Koyejo S , Mohamed S , Agarwal A , Belgrave D , Cho K , Oh A , eds. Advances in Neural Information Processing Systems 35: 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Vol 35. Curran Associates, Inc.; 2022:24824-24837."},{"key":"2025092903550797000_2025.05.23.25328115v2.45","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2018.3763"},{"key":"2025092903550797000_2025.05.23.25328115v2.46","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2404.17218"},{"key":"2025092903550797000_2025.05.23.25328115v2.47","unstructured":"Poibeau T , Messiant C. Do we Still Need Gold Standards for Evaluation? In: Calzolari N , Choukri K , Maegaard B , et al. , eds. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC \u201808). European Language Resources Association (ELRA); 2008. http:\/\/www.lrec-conf.org\/proceedings\/lrec2008\/pdf\/144_paper.pdf"},{"key":"2025092903550797000_2025.05.23.25328115v2.48","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2305.10284"},{"key":"2025092903550797000_2025.05.23.25328115v2.49","unstructured":"Hoffman K , Salerno S , Afiaz A , Leek JT , McCormick TH . Do We Really Even Need Data? arXiv. Preprint posted online February 2, 2024. Accessed August 29, 2024. http:\/\/arxiv.org\/abs\/2401.08702"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.05.23.25328115","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:36:49Z","timestamp":1768509409000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2025.05.23.25328115"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,25]]},"references-count":49,"URL":"https:\/\/doi.org\/10.1101\/2025.05.23.25328115","relation":{},"subject":[],"published":{"date-parts":[[2025,5,25]]},"subtype":"preprint"}}