{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,23]],"date-time":"2026-04-23T13:01:05Z","timestamp":1776949265606,"version":"3.51.4"},"reference-count":19,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T00:00:00Z","timestamp":1719273600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R00LM014097-02"],"award-info":[{"award-number":["R00LM014097-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AG062499-01"],"award-info":[{"award-number":["R01AG062499-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM013995-01"],"award-info":[{"award-number":["R01LM013995-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>This study aims to investigate the feasibility of using Large Language Models (LLMs) to engage with patients at the time they are drafting a question to their healthcare providers, and generate pertinent follow-up questions that the patient can answer before sending their message, with the goal of ensuring that their healthcare provider receives all the information they need to safely and accurately answer the patient\u2019s question, eliminating back-and-forth messaging, and the associated delays and frustrations.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We collected a dataset of patient messages sent between January 1, 2022 to March 7, 2023 at Vanderbilt University Medical Center. Two internal medicine physicians identified 7 common scenarios. We used 3 LLMs to generate follow-up questions: (1) Comprehensive LLM Artificial Intelligence Responder (CLAIR): a locally fine-tuned LLM, (2) GPT4 with a simple prompt, and (3) GPT4 with a complex prompt. Five physicians rated them with the actual follow-ups written by healthcare providers on clarity, completeness, conciseness, and utility.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>For five scenarios, our CLAIR model had the best performance. The GPT4 model received higher scores for utility and completeness but lower scores for clarity and conciseness. CLAIR generated follow-up questions with similar clarity and conciseness as the actual follow-ups written by healthcare providers, with higher utility than healthcare providers and GPT4, and lower completeness than GPT4, but better than healthcare providers.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>LLMs can generate follow-up patient messages designed to clarify a medical question that compares favorably to those generated by healthcare providers.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae142","type":"journal-article","created":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T21:17:03Z","timestamp":1719350223000},"page":"1665-1670","source":"Crossref","is-referenced-by-count":29,"title":["Using large language model to guide patients to create efficient and comprehensive clinical care message"],"prefix":"10.1093","volume":"31","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212, United States"}]},{"given":"Aileen P","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"Mccoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}]},{"given":"Sean S","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"Julian Z","family":"Genkins","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"Josh F","family":"Peterson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"Yaa A","family":"Kumah-Crystal","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Pediatric Endocrinology, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"William","family":"Martinez","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"Babatunde","family":"Carew","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"Dara","family":"Mize","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]},{"given":"Bryan","family":"Steitz","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,6,25]]},"reference":[{"issue":"3","key":"2024071907530578500_ocae142-B1","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1093\/jamia\/ocab268","article-title":"Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use","volume":"29","author":"Holmgren","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"1","key":"2024071907530578500_ocae142-B2","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1186\/s12875-019-0948-1","article-title":"Patient portal messaging for care coordination: A qualitative study of perspectives of experienced users with chronic conditions","volume":"20","author":"Hefner","year":"2019","journal-title":"BMC Fam Pract"},{"issue":"11-12","key":"2024071907530578500_ocae142-B3","doi-asserted-by":"crossref","first-page":"e383\u201390","DOI":"10.1093\/milmed\/usy094","article-title":"Communication preferences and satisfaction of secure messaging among patients and providers in the military healthcare system","volume":"183","author":"Hernandez","year":"2018","journal-title":"Mil Med"},{"issue":"4","key":"2024071907530578500_ocae142-B4","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1001\/jamainternmed.2016.0209","article-title":"The burden of inbox notifications in commercial electronic health records","volume":"176","author":"Murphy","year":"2016","journal-title":"JAMA Intern Med"},{"issue":"7","key":"2024071907530578500_ocae142-B5","doi-asserted-by":"crossref","first-page":"1073","DOI":"10.1377\/hlthaff.2018.05509","article-title":"Physicians\u2019 well-being linked to in-basket messages generated by algorithms in electronic health records","volume":"38","author":"Tai-Seale","year":"2019","journal-title":"Health Aff"},{"issue":"4","key":"2024071907530578500_ocae142-B6","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1093\/jamia\/ocz220","article-title":"Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians","volume":"27","author":"Adler-Milstein","year":"2020","journal-title":"J Am Med Informatics Assoc"},{"issue":"1","key":"2024071907530578500_ocae142-B7","doi-asserted-by":"crossref","first-page":"e17273","DOI":"10.2196\/17273","article-title":"Characterizing patient-clinician communication in secure medical messages: retrospective study","volume":"24","author":"Huang","year":"2022","journal-title":"J Med Internet Res"},{"issue":"6","key":"2024071907530578500_ocae142-B8","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence Chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"issue":"6","key":"2024071907530578500_ocae142-B9","doi-asserted-by":"crossref","first-page":"1367","DOI":"10.1093\/jamia\/ocae052","article-title":"Leveraging large language models for generating responses to patient messages\u2014a subjective analysis","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Informatics Assoc"},{"issue":"4","key":"2024071907530578500_ocae142-B10","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1093\/jamiaopen\/ooz039","article-title":"Policies and procedures governing patient portal use at an Academic Medical Center","volume":"2","author":"Steitz","year":"2019","journal-title":"JAMIA Open"},{"key":"2024071907530578500_ocae142-B11","article-title":"LoRA: low-rank adaptation of large language models","author":"Hu","year":"2021"},{"key":"2024071907530578500_ocae142-B12","author":"Touvron","year":"2023"},{"key":"2024071907530578500_ocae142-B13","author":"Han","year":"2023"},{"issue":"260","key":"2024071907530578500_ocae142-B14","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","article-title":"Use of ranks in one-criterion variance analysis","volume":"47","author":"Kruskal","year":"1952","journal-title":"J Am Stat Assoc"},{"issue":"293","key":"2024071907530578500_ocae142-B15","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","article-title":"Multiple comparisons among means","volume":"56","author":"Dunn","year":"1961","journal-title":"J Am Stat Assoc"},{"issue":"2","key":"2024071907530578500_ocae142-B16","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"2024071907530578500_ocae142-B17","author":"Adams","year":"2023"},{"key":"2024071907530578500_ocae142-B18","doi-asserted-by":"crossref","first-page":"e48451","DOI":"10.2196\/48451","article-title":"Potential use of ChatGPT in responding to patient questions and creating patient resources","volume":"7","author":"Reynolds","year":"2024","journal-title":"JMIR Dermatol"},{"issue":"6","key":"2024071907530578500_ocae142-B19","doi-asserted-by":"crossref","first-page":"1388","DOI":"10.1093\/jamia\/ocae041","article-title":"Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/8\/1665\/58591249\/ocae142.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/8\/1665\/58591249\/ocae142.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T07:53:18Z","timestamp":1721375598000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/8\/1665\/7699038"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,25]]},"references-count":19,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,6,25]]},"published-print":{"date-parts":[[2024,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae142","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,8]]},"published":{"date-parts":[[2024,6,25]]}}}