{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T06:26:29Z","timestamp":1782800789269,"version":"3.54.5"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>Large Language Models (LLMs) have been proposed as a solution to address high volumes of Patient Medical Advice Requests (PMARs). This study addresses whether LLMs can generate high quality draft responses to PMARs that satisfies both patients and clinicians with prompt engineering.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We designed a novel human-involved iterative processes to train and validate prompts to LLM in creating appropriate responses to PMARs. GPT-4 was used to generate response to the messages. We updated the prompts, and evaluated both clinician and patient acceptance of LLM-generated draft responses at each iteration, and tested the optimized prompt on independent validation data sets. The optimized prompt was implemented in the electronic health record production environment and tested by 69 primary care clinicians.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>After 3 iterations of prompt engineering, physician acceptance of draft suitability increased from 62% to 84% (P\u2009&amp;lt;.001) in the validation dataset (N\u2009=\u2009200), and 74% of drafts in the test dataset were rated as \u201chelpful.\u201d Patients also noted significantly increased favorability of message tone (78%) and overall quality (80%) for the optimized prompt compared to the original prompt in the training dataset, patients were unable to differentiate human and LLM-generated draft PMAR responses for 76% of the messages, in contrast to the earlier preference for human-generated responses. Majority (72%) of clinicians believed it can reduce cognitive load in dealing with InBasket messages.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion and Conclusion<\/jats:title>\n                  <jats:p>Informed by clinician and patient feedback synergistically, tuning in LLM prompt alone can be effective in creating clinically relevant and useful draft responses to PMARs.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae172","type":"journal-article","created":{"date-parts":[[2024,7,20]],"date-time":"2024-07-20T01:19:36Z","timestamp":1721438376000},"page":"2263-2270","source":"Crossref","is-referenced-by-count":38,"title":["Prompt engineering on leveraging large language models in generating response to InBasket messages"],"prefix":"10.1093","volume":"31","author":[{"given":"Sherry","family":"Yan","sequence":"first","affiliation":[{"name":"Center for Health Systems Research, Sutter Health , Walnut Creek, CA 94596, United States"},{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wendi","family":"Knapp","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"},{"name":"Palo Alto Foundation Medical Group, Sutter Health , Palo Alto, CA 94301, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Leong","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"},{"name":"Sutter Digital Engagement, Sutter Health , Sacramento, CA 95833, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarira","family":"Kadkhodazadeh","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"},{"name":"Palo Alto Foundation Medical Group, Sutter Health , Palo Alto, CA 94301, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Souvik","family":"Das","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"},{"name":"Sutter Data and Analytics Department, Sutter Health , Sacramento, CA 95833, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Veena G","family":"Jones","sequence":"additional","affiliation":[{"name":"Palo Alto Foundation Medical Group, Sutter Health , Palo Alto, CA 94301, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Clark","sequence":"additional","affiliation":[{"name":"Palo Alto Foundation Medical Group, Sutter Health , Palo Alto, CA 94301, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Grattendick","sequence":"additional","affiliation":[{"name":"Sutter Medical Group, Sutter Health , Roseville, CA 95661, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Chen","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa","family":"Hladik","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lawrence","family":"Fagan","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"},{"name":"Patient Family Advisory Council, Sutter Health , Palo Alto, CA 94301, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Albert","family":"Chan","sequence":"additional","affiliation":[{"name":"Sutter Health Data Science Team, Sutter Health , Sacramento, CA 95833, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,7,19]]},"reference":[{"key":"2024092007531796400_ocae172-B1","author":"The Centers for Medicare & Medicaid Services (CMS)"},{"issue":"10","key":"2024092007531796400_ocae172-B2","doi-asserted-by":"publisher","first-page":"e2131490","DOI":"10.1001\/jamanetworkopen.2021.31490","article-title":"Trends in electronic health record inbox messaging during the COVID-19 pandemic in an ambulatory practice network in New England","volume":"4","author":"Nath","year":"2021","journal-title":"JAMA Netw Open"},{"issue":"7","key":"2024092007531796400_ocae172-B3","doi-asserted-by":"publisher","first-page":"1073","DOI":"10.1377\/hlthaff.2018.05509","article-title":"Physicians\u2019 well-being linked to in-basket messages generated by algorithms in electronic health records","volume":"38","author":"Tai-Seale","year":"2019","journal-title":"Health Aff (Millwood)"},{"issue":"4","key":"2024092007531796400_ocae172-B4","doi-asserted-by":"publisher","first-page":"655","DOI":"10.1377\/hlthaff.2016.0811","article-title":"Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine","volume":"36","author":"Tai-Seale","year":"2017","journal-title":"Health Aff (Millwood)"},{"issue":"4","key":"2024092007531796400_ocae172-B5","doi-asserted-by":"publisher","first-page":"479","DOI":"10.4300\/JGME-D-16-00123.1","article-title":"Electronic health record effects on work-life balance and burnout within the I3 population collaborative","volume":"9","author":"Robertson","year":"2017","journal-title":"J Grad Med Educ"},{"issue":"5","key":"2024092007531796400_ocae172-B6","doi-asserted-by":"publisher","DOI":"10.1056\/CAT.22.0438","article-title":"In-basket reduction: a multiyear pragmatic approach to lessen the work burden of primary care physicians","volume":"4","author":"Fogg Jane","journal-title":"NEJM Catal"},{"key":"2024092007531796400_ocae172-B7","doi-asserted-by":"publisher","first-page":"101304","DOI":"10.1016\/j.imu.2023.101304","article-title":"Evaluating large language models for use in healthcare: a framework for translational value assessment","volume":"41","author":"Reddy","year":"2023","journal-title":"Inform Med Unlocked"},{"issue":"9","key":"2024092007531796400_ocae172-B8","doi-asserted-by":"publisher","first-page":"e233909","DOI":"10.1001\/jamahealthforum.2023.3909","article-title":"How to safely integrate large language models into health care","volume":"4","author":"Gottlieb","year":"2023","journal-title":"JAMA Health Forum"},{"issue":"12","key":"2024092007531796400_ocae172-B9","doi-asserted-by":"publisher","first-page":"2969","DOI":"10.1038\/s41591-023-02700-1","article-title":"Six ways large language models are changing healthcare","volume":"29","author":"Webster","year":"2023","journal-title":"Nat Med"},{"issue":"4","key":"2024092007531796400_ocae172-B10","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1001\/jamainternmed.2023.7965","article-title":"Using artificial intelligence to improve primary care for patients and clinicians","volume":"184","author":"Sarkar","year":"2024","journal-title":"JAMA Intern Med"},{"issue":"6654","key":"2024092007531796400_ocae172-B11","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1126\/science.adh2586","article-title":"Experimental evidence on the productivity effects of generative artificial intelligence","volume":"381","author":"Noy","year":"2023","journal-title":"Science"},{"issue":"13","key":"2024092007531796400_ocae172-B12","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"issue":"9","key":"2024092007531796400_ocae172-B13","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and adoption of large language models in medicine","volume":"330","author":"Shah","year":"2023","journal-title":"JAMA"},{"issue":"12","key":"2024092007531796400_ocae172-B14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024092007531796400_ocae172-B15","first-page":"675","volume-title":"Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Bang","year":"2023"},{"key":"2024092007531796400_ocae172-B16","first-page":"314","author":"Pal"},{"issue":"6","key":"2024092007531796400_ocae172-B17","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"issue":"3","key":"2024092007531796400_ocae172-B18","doi-asserted-by":"publisher","first-page":"e243201","DOI":"10.1001\/jamanetworkopen.2024.3201","article-title":"Artificial intelligence\u2013generated draft replies to patient inbox messages","volume":"7","author":"Garcia","year":"2024","journal-title":"JAMA Netw Open"},{"key":"2024092007531796400_ocae172-B19","author":"White","year":"2023"},{"issue":"3","key":"2024092007531796400_ocae172-B20","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1016\/j.pec.2008.11.015","article-title":"How does communication heal? Pathways linking clinician\u2013patient communication to health outcomes","volume":"74","author":"Street","year":"2009","journal-title":"Patient Educ Couns"},{"issue":"6","key":"2024092007531796400_ocae172-B21","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1001\/jama.2019.21579","article-title":"Artificial intelligence in health care: a report from the National Academy of Medicine","volume":"323","author":"Matheny","year":"2020","journal-title":"JAMA"},{"issue":"1","key":"2024092007531796400_ocae172-B22","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1186\/s13000-024-01464-7","article-title":"Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology\u2014a recent scoping review","volume":"19","author":"Ullah","year":"2024","journal-title":"Diagn Pathol"},{"issue":"15","key":"2024092007531796400_ocae172-B23","doi-asserted-by":"publisher","first-page":"4002","DOI":"10.1007\/s11606-022-07766-0","article-title":"The electronic health record inbox: recommendations for relief","volume":"37","author":"Sinsky","year":"2022","journal-title":"J Gen Intern Med"},{"key":"2024092007531796400_ocae172-B24","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/B978-0-12-387691-1.00002-8","volume-title":"Psychology of Learning and Motivation","author":"Sweller","year":"2011"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/10\/2263\/59206379\/ocae172.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/10\/2263\/59206379\/ocae172.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T07:53:37Z","timestamp":1726818817000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/10\/2263\/7717344"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,19]]},"references-count":24,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,7,19]]},"published-print":{"date-parts":[[2024,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae172","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,10]]},"published":{"date-parts":[[2024,7,19]]}}}