{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T16:11:24Z","timestamp":1774541484485,"version":"3.50.1"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T00:00:00Z","timestamp":1710720000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R00LM014097-01"],"award-info":[{"award-number":["R00LM014097-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AG062499-01"],"award-info":[{"award-number":["R01AG062499-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM013995-01"],"award-info":[{"award-number":["R01LM013995-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,5,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The dataset consisted of 499\u00a0794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider\u2019s responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT\u2019s responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae052","type":"journal-article","created":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T19:56:55Z","timestamp":1710791815000},"page":"1367-1379","source":"Crossref","is-referenced-by-count":56,"title":["Leveraging large language models for generating responses to patient messages\u2014a subjective analysis"],"prefix":"10.1093","volume":"31","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Aileen P","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Babatunde","family":"Carew","sequence":"additional","affiliation":[{"name":"Department of General Internal Medicine and Public Health, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Julian Z","family":"Genkins","sequence":"additional","affiliation":[{"name":"Department of Medicine, Stanford University , Stanford, CA 94304, United States"}]},{"given":"Sean S","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Josh F","family":"Peterson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Bryan","family":"Steitz","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,3,18]]},"reference":[{"key":"2024052019465548600_ocae052-B1","doi-asserted-by":"crossref","first-page":"104037","DOI":"10.1016\/j.ijmedinf.2019.104037","article-title":"Quantifying the competitiveness of the electronic health record market and its implications for interoperability","volume":"136","author":"Sorace","year":"2020","journal-title":"Int J Med Inform"},{"issue":"3","key":"2024052019465548600_ocae052-B2","doi-asserted-by":"crossref","first-page":"e109","DOI":"10.2196\/jmir.7851","article-title":"Growth dynamics of patient-provider internet communication: trend analysis using the health information national trends survey (2003 to 2013)","volume":"20","author":"Tarver","year":"2018","journal-title":"J Med Internet Res"},{"issue":"5","key":"2024052019465548600_ocae052-B3","doi-asserted-by":"crossref","first-page":"923","DOI":"10.1093\/jamia\/ocaa229","article-title":"Physicians\u2019 electronic inbox work patterns and factors associated with high inbox work duration","volume":"28","author":"Akbar","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2024052019465548600_ocae052-B4","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"Arndt","year":"2017","journal-title":"Ann Fam Med"},{"issue":"10","key":"2024052019465548600_ocae052-B5","doi-asserted-by":"crossref","first-page":"e2129553","DOI":"10.1001\/jamanetworkopen.2021.29553","article-title":"Association of immediate release of test results to patients with implications for clinical workflow","volume":"4","author":"Steitz","year":"2021","journal-title":"JAMA Netw Open"},{"issue":"15","key":"2024052019465548600_ocae052-B6","doi-asserted-by":"crossref","first-page":"4002","DOI":"10.1007\/s11606-022-07766-0","article-title":"The electronic health record inbox: recommendations for relief","volume":"37","author":"Sinsky","year":"2022","journal-title":"J Gen Intern Med"},{"issue":"3","key":"2024052019465548600_ocae052-B7","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1093\/jamia\/ocab268","article-title":"Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use","volume":"29","author":"Holmgren","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"2024052019465548600_ocae052-B8","doi-asserted-by":"crossref","first-page":"e1918287","DOI":"10.1001\/jamanetworkopen.2019.18287","article-title":"Primary care physicians\u2019 experiences with and strategies for managing electronic messages","volume":"2","author":"Lieu","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"4","key":"2024052019465548600_ocae052-B9","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1093\/jamia\/ocz220","article-title":"Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians","volume":"27","author":"Adler-Milstein","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2024052019465548600_ocae052-B10","doi-asserted-by":"crossref","first-page":"102274","DOI":"10.1016\/j.lindif.2023.102274","article-title":"ChatGPT for good? On opportunities and challenges of large language models for education","volume":"103","author":"Kasneci","year":"2023","journal-title":"Learn Individ Differ"},{"key":"2024052019465548600_ocae052-B11","author":"ChatGPT: Optimizing Language Models for Dialogue","year":"2022"},{"key":"2024052019465548600_ocae052-B12","doi-asserted-by":"crossref","first-page":"e48568","DOI":"10.2196\/48568","article-title":"Utility of ChatGPT in clinical practice","volume":"25","author":"Liu","year":"2023","journal-title":"J Med Internet Res"},{"issue":"6","key":"2024052019465548600_ocae052-B13","doi-asserted-by":"crossref","first-page":"102064","DOI":"10.1016\/j.outlook.2023.102064","article-title":"The application of chat generative pre-trained transformer in nursing education","volume":"71","author":"Liu","year":"2023","journal-title":"Nurs Outlook"},{"issue":"2","key":"2024052019465548600_ocae052-B14","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"7","key":"2024052019465548600_ocae052-B15","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Inform Assoc"},{"issue":"9","key":"2024052019465548600_ocae052-B16","doi-asserted-by":"crossref","first-page":"1558","DOI":"10.1093\/jamia\/ocad104","article-title":"ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?","volume":"30","author":"Kumah-Crystal","year":"2023","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024052019465548600_ocae052-B17","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence Chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2024052019465548600_ocae052-B18","author":"Min","year":"2021"},{"key":"2024052019465548600_ocae052-B19","author":"Li","year":"2023"},{"issue":"1","key":"2024052019465548600_ocae052-B20","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/s10916-023-01925-4","article-title":"Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios","volume":"47","author":"Cascella","year":"2023","journal-title":"J Med Syst"},{"issue":"8","key":"2024052019465548600_ocae052-B21","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2024052019465548600_ocae052-B22","doi-asserted-by":"crossref","first-page":"e48009","DOI":"10.2196\/48009","article-title":"Ethical considerations of using ChatGPT in health care","volume":"25","author":"Wang","year":"2023","journal-title":"J Med Internet Res"},{"issue":"2","key":"2024052019465548600_ocae052-B23","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1197\/jamia.M3092","article-title":"A \u2018Fundamental Theorem\u2019 of biomedical informatics","volume":"16","author":"Friedman","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2024052019465548600_ocae052-B24","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1093\/jamia\/ocac219","article-title":"Automated deidentification of radiology reports combining transformer and \u201chide in plain sight\u201d rule-based methods","volume":"30","author":"Chambon","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2024052019465548600_ocae052-B25","author":"Touvron","year":"2023"},{"key":"2024052019465548600_ocae052-B26","author":"Hu","year":"2021"},{"key":"2024052019465548600_ocae052-B27","author":"Peng","year":"2023"},{"key":"2024052019465548600_ocae052-B28","author":"Taori","year":"2023"},{"issue":"11","key":"2024052019465548600_ocae052-B29","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1089\/tmj.2019.0192","article-title":"Look who\u2019s talking: application of a theory-based taxonomy to patient\u2013clinician E-mail messages","volume":"26","author":"Heisey-Grove","year":"2020","journal-title":"Telemed e-Health"},{"key":"2024052019465548600_ocae052-B30","author":"Abid","year":"2019"},{"issue":"2","key":"2024052019465548600_ocae052-B31","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1016\/j.jbi.2008.08.010","article-title":"Research electronic data capture (REDCap)\u2014a metadata-driven methodology and workflow process for providing translational research informatics support","volume":"42","author":"Harris","year":"2009","journal-title":"J Biomed Inform"},{"key":"2024052019465548600_ocae052-B32","author":"Zhang","year":"2019"},{"issue":"2","key":"2024052019465548600_ocae052-B33","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"2024052019465548600_ocae052-B34","author":"Zhou","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1367\/57769034\/ocae052.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1367\/57769034\/ocae052.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,20]],"date-time":"2024-05-20T19:48:02Z","timestamp":1716234482000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/6\/1367\/7630962"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,18]]},"references-count":34,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,3,18]]},"published-print":{"date-parts":[[2024,5,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae052","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,6,1]]},"published":{"date-parts":[[2024,3,18]]}}}