{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T04:54:12Z","timestamp":1774500852403,"version":"3.50.1"},"reference-count":23,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R00LM014097-02"],"award-info":[{"award-number":["R00LM014097-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AG062499-01"],"award-info":[{"award-number":["R01AG062499-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM013995-01"],"award-info":[{"award-number":["R01LM013995-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,5,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4\u2009\u00b1\u20091.2, human: 2.7\u2009\u00b1\u20091.2, P\u2009=\u2009.001).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>End-user comments provide clinicians\u2019 immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae041","type":"journal-article","created":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T20:47:05Z","timestamp":1709844425000},"page":"1388-1396","source":"Crossref","is-referenced-by-count":24,"title":["Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support"],"prefix":"10.1093","volume":"31","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Aileen P","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1941-1817","authenticated-orcid":false,"given":"Scott D","family":"Nelson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Sean S","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Hasan B","family":"Ahmad","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, WA 98195, United States"}]},{"given":"Sabrina E","family":"Carro","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"Jacob","family":"Franklin","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"given":"James","family":"Brogan","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37212, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37212, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,3,7]]},"reference":[{"key":"2024052019465090200_ocae041-B1","first-page":"1","article-title":"Hospitals\u2019 use of electronic health records data, 2015-2017","volume":"46","author":"Parasrampuria","year":"2019","journal-title":"ONC Data Br"},{"issue":"3","key":"2024052019465090200_ocae041-B2","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1136\/amiajnl-2011-000113","article-title":"Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems","volume":"18","author":"Wright","year":"2011","journal-title":"J Am Med Informatics Assoc"},{"issue":"3","key":"2024052019465090200_ocae041-B3","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.cvdhj.2020.11.001","article-title":"Rapid review: identification of digital health interventions in atherosclerotic-related cardiovascular disease populations to address racial, ethnic, and socioeconomic health disparities","volume":"1","author":"Thomas Craig","year":"2020","journal-title":"Cardiovasc Digit Health J"},{"issue":"3","key":"2024052019465090200_ocae041-B4","doi-asserted-by":"crossref","first-page":"331","DOI":"10.4338\/ACI-2010-05-RA-0031","article-title":"Best practices in clinical decision support: the case of preventive care reminders","volume":"1","author":"Wright","year":"2010","journal-title":"Appl Clin Inform"},{"issue":"1","key":"2024052019465090200_ocae041-B5","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1055\/s-0043-1768722","article-title":"The impact of clinical decision support on health disparities and the digital divide","volume":"32","author":"Douthit","year":"2023","journal-title":"Yearb Med Inform"},{"key":"2024052019465090200_ocae041-B6","doi-asserted-by":"crossref","DOI":"10.4324\/9781498757461","volume-title":"Improving Outcomes with Clinical Decision Support: An Implementer\u2019s Guide","author":"Osheroff","year":"2012","edition":"2nd ed"},{"issue":"4","key":"2024052019465090200_ocae041-B7","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1016\/j.ijmedinf.2013.12.006","article-title":"What, if all alerts were specific\u2014estimating the potential impact on drug interaction alert burden","volume":"83","author":"Seidling","year":"2014","journal-title":"Int J Med Inform"},{"issue":"2","key":"2024052019465090200_ocae041-B8","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1197\/jamia.M1809","article-title":"Overriding of drug safety alerts in computerized physician order entry","volume":"13","author":"van der Sijs","year":"2006","journal-title":"J Am Med Informatics Assoc"},{"issue":"11","key":"2024052019465090200_ocae041-B9","doi-asserted-by":"crossref","first-page":"1868","DOI":"10.1007\/s11606-018-4415-9","article-title":"Reduced effectiveness of interruptive drug-drug interaction alerts after conversion to a commercial electronic health record","volume":"33","author":"Wright","year":"2018","journal-title":"J Gen Intern Med"},{"issue":"6","key":"2024052019465090200_ocae041-B10","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1093\/jamia\/ocac027","article-title":"Clinician collaboration to improve clinical decision support: the Clickbusters initiative","volume":"29","author":"McCoy","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"5","key":"2024052019465090200_ocae041-B11","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1093\/jamia\/ocab292","article-title":"The potential for leveraging machine learning to filter medication alerts","volume":"29","author":"Liu","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"7","key":"2024052019465090200_ocae041-B12","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Informatics Assoc"},{"issue":"5","key":"2024052019465090200_ocae041-B13","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1093\/jamia\/ocx106","article-title":"Clinical decision support alert malfunctions: analysis and empirically derived taxonomy","volume":"25","author":"Wright","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024052019465090200_ocae041-B14","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1093\/jamia\/ocy139","article-title":"Cranky comments: detecting clinical decision support malfunctions through free-text override reasons","volume":"26","author":"Aaron","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2024052019465090200_ocae041-B15","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1136\/amiajnl-2012-001089","article-title":"Drug\u2014Drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records","volume":"20","author":"Phansalkar","year":"2013","journal-title":"J Am Med Informatics Assoc"},{"key":"2024052019465090200_ocae041-B16","author":"Zhang","year":"2023"},{"key":"2024052019465090200_ocae041-B17","author":"Liu","year":"2023"},{"key":"2024052019465090200_ocae041-B18","author":"Zhang","year":"2023"},{"key":"2024052019465090200_ocae041-B19","author":"Ma","year":"2023"},{"issue":"4","key":"2024052019465090200_ocae041-B20","first-page":"e38249","article-title":"Enhancing expert panel discussions in pediatric palliative care: innovative scenario development and summarization with ChatGPT-4","volume":"15","author":"Almazyad","year":"2023","journal-title":"Cureus"},{"key":"2024052019465090200_ocae041-B21","doi-asserted-by":"crossref","first-page":"e48568","DOI":"10.2196\/48568","article-title":"Utility of ChatGPT in clinical practice","volume":"25","author":"Liu","year":"2023","journal-title":"J Med Internet Res"},{"key":"2024052019465090200_ocae041-B22","doi-asserted-by":"crossref","first-page":"e50638","DOI":"10.2196\/50638","article-title":"Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial","volume":"25","author":"Mesk\u00f3","year":"2023","journal-title":"J Med Internet Res"},{"issue":"2","key":"2024052019465090200_ocae041-B23","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1388\/57769023\/ocae041.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1388\/57769023\/ocae041.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,20]],"date-time":"2024-05-20T19:47:58Z","timestamp":1716234478000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/6\/1388\/7624148"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,7]]},"references-count":23,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,3,7]]},"published-print":{"date-parts":[[2024,5,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae041","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,6,1]]},"published":{"date-parts":[[2024,3,7]]}}}