{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T15:04:41Z","timestamp":1773414281312,"version":"3.50.1"},"reference-count":21,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T00:00:00Z","timestamp":1749686400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>The integration of Large Language Models (LLMs) in Electronic Health Records (EHRs) has the potential to reduce administrative burden. Validating these tools in real-world clinical settings is essential for responsible implementation. In this study, the effect of implementing LLM-generated draft responses to patient questions in our EHR is evaluated with regard to adoption, use and potential time savings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Material and methods<\/jats:title><jats:p>Physicians across 14 medical specialties in a non-English large academic hospital were invited to use LLM-generated draft replies during this prospective observational clinical cohort study of 16 weeks, choosing either the drafted or a blank reply. The adoption rate, the level of adjustments to the initial drafted responses compared to the final sent messages (using ROUGE-1 and BLEU-1 natural language processing scores), and the time spent on these adjustments were analyzed.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>A total of 919 messages by 100 physicians were evaluated. Clinicians used the LLM draft in 58% of replies. Of these, 43% used a large part of the suggested text for the final answer (\u226510% match drafted responses: ROUGE-1: 86% similarity, vs. blank replies: ROUGE-1: 16%). Total response time did not significantly different when using a blank reply compared to using a drafted reply with \u226510% match (157 vs. 153\u2005s, <jats:italic>p<\/jats:italic>\u2009=\u20090.69).<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>General adoption of LLM-generated draft responses to patient messages was 58%, although the level of adjustments on the drafted message varied widely between medical specialties. This implicates safe use in a non-English, tertiary setting. The current implementation has not yet resulted in time savings, but a learning curve can be expected.<\/jats:p><\/jats:sec><jats:sec><jats:title>Registration number<\/jats:title><jats:p>19035.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fdgth.2025.1588143","type":"journal-article","created":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T05:21:17Z","timestamp":1749705677000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["AI-generated draft replies to patient messages: exploring effects of implementation"],"prefix":"10.3389","volume":"7","author":[{"given":"Charlotte M. H. H. T.","family":"Bootsma-Robroeks","sequence":"first","affiliation":[]},{"given":"Jessica D.","family":"Workum","sequence":"additional","affiliation":[]},{"given":"Stephanie C. E.","family":"Schuit","sequence":"additional","affiliation":[]},{"given":"Anne","family":"Hoekman","sequence":"additional","affiliation":[]},{"given":"Tarannom","family":"Mehri","sequence":"additional","affiliation":[]},{"given":"Job N.","family":"Doornberg","sequence":"additional","affiliation":[]},{"given":"Tom P.","family":"van der Laan","sequence":"additional","affiliation":[]},{"given":"Rosanne C.","family":"Schoonbeek","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2025,6,12]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"923","DOI":"10.1093\/jamia\/ocaa229","article-title":"Physicians\u2019 electronic inbox work patterns and factors associated with high inbox work duration","volume":"28","author":"Akbar","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"B2","doi-asserted-by":"publisher","first-page":"e48583","DOI":"10.2196\/48583","article-title":"Characterizing the patterns of electronic health record-integrated secure messaging use: cross-sectional study","volume":"25","author":"Baratta","year":"2023","journal-title":"J Med Internet Res"},{"key":"B3","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1097\/ACM.0000000000001461","article-title":"The impact of administrative burden on academic physicians: results of a hospital-wide physician survey","volume":"92","author":"Rao","year":"2017","journal-title":"Acad Med"},{"key":"B4","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1093\/jamia\/ocab268","article-title":"Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use","volume":"29","author":"Holmgren","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"B5","doi-asserted-by":"publisher","first-page":"827","DOI":"10.1016\/j.healthpol.2018.05.014","article-title":"The impact of electronic health record systems on clinical documentation times: a systematic review","volume":"122","author":"Baumann","year":"2018","journal-title":"Health Policy"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"B7","doi-asserted-by":"publisher","first-page":"e2422399","DOI":"10.1001\/jamanetworkopen.2024.22399","article-title":"Large language model-based responses to patients\u2019 in-basket messages","volume":"7","author":"Small","year":"2024","journal-title":"JAMA Netw Open"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"B9","doi-asserted-by":"publisher","first-page":"E246565","DOI":"10.1001\/jamanetworkopen.2024.6565","article-title":"AI-generated draft replies integrated into health records and physicians\u2019 electronic communication","volume":"7","author":"Tai-Seale","year":"2024","journal-title":"JAMA Netw Open"},{"key":"B10","doi-asserted-by":"publisher","first-page":"e244630","DOI":"10.1001\/jamanetworkopen.2024.4630","article-title":"Quality of large language model responses to radiation oncology patient care questions","volume":"7","author":"Yalamanchili","year":"2024","journal-title":"JAMA Netw Open"},{"key":"B11","doi-asserted-by":"publisher","first-page":"E243201","DOI":"10.1001\/jamanetworkopen.2024.3201","article-title":"Artificial intelligence-generated draft replies to patient inbox messages","volume":"7","author":"Garcia","year":"2024","journal-title":"JAMA Netw Open"},{"key":"B12","doi-asserted-by":"crossref","DOI":"10.1101\/2024.04.15.24305869","article-title":"A systematic review of testing and evaluation of healthcare applications of large language models (LLMs)","volume-title":"medRxiv","author":"Bedi","year":"2024"},{"key":"B13","volume-title":"arXiv:2404.11553","author":"Li","year":"2024"},{"key":"B14","doi-asserted-by":"publisher","first-page":"e50638","DOI":"10.2196\/50638","article-title":"Prompt engineering as an important emerging skill for medical professionals: tutorial","volume":"25","author":"Mesk\u00f3","year":"2023","journal-title":"J Med Internet Res"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2263","DOI":"10.1093\/jamia\/ocae172","article-title":"Prompt engineering on leveraging large language models in generating response to InBasket messages","volume":"31","author":"Yan","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"B16","first-page":"74","article-title":"ROUGE: a package for automatic evaluation of summaries","volume-title":"Proceedings of the ACL Workshop: Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"B17","first-page":"311","article-title":"BLEU: a method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL '02)","author":"Papineni","year":"2002"},{"key":"B18","volume-title":"Twelve years of measuring linguistic diversity in the 396 Internet: balance and perspectives","author":"Pimienta","year":"2009"},{"key":"B19","doi-asserted-by":"crossref","DOI":"10.1109\/BigData62323.2024.10825307","article-title":"Assessing empathy in large language models with real-world physician-patient interactions","volume-title":"arXiv:2405.16402","author":"Luo","year":"2024"},{"key":"B20","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1367","DOI":"10.1093\/jamia\/ocae052","article-title":"Leveraging large language models for generating responses to patient messages\u2014a subjective analysis","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inform Assoc"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2025.1588143\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T05:21:19Z","timestamp":1749705679000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2025.1588143\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,12]]},"references-count":21,"alternative-id":["10.3389\/fdgth.2025.1588143"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2025.1588143","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,12]]},"article-number":"1588143"}}