{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T16:14:19Z","timestamp":1784996059872,"version":"3.55.0"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T00:00:00Z","timestamp":1708560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R00LM014097-02"],"award-info":[{"award-number":["R00LM014097-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AG062499-01"],"award-info":[{"award-number":["R01AG062499-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01LM013995-01"],"award-info":[{"award-number":["R01LM013995-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,4,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert\u2019s historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The final dataset included 2\u200a991\u200a823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278\u200a807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae019","type":"journal-article","created":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T01:50:50Z","timestamp":1708566650000},"page":"968-974","source":"Crossref","is-referenced-by-count":37,"title":["Leveraging explainable artificial intelligence to optimize clinical decision support"],"prefix":"10.1093","volume":"31","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josh F","family":"Peterson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas A","family":"Lasko","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Computer Science, Vanderbilt University , Nashville, TN 37212, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5811-8915","authenticated-orcid":false,"given":"Dean F","family":"Sittig","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, University of Texas Health Science Center , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1941-1817","authenticated-orcid":false,"given":"Scott D","family":"Nelson","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Andrews","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lorraine","family":"Patterson","sequence":"additional","affiliation":[{"name":"HeathIT, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheryl M","family":"Cobb","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Sciences, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Mulherin","sequence":"additional","affiliation":[{"name":"HeathIT, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Colleen T","family":"Morton","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203, United States"},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,2,22]]},"reference":[{"key":"2024040320013586400_ocae019-B1","doi-asserted-by":"crossref","first-page":"104037","DOI":"10.1016\/j.ijmedinf.2019.104037","article-title":"Quantifying the competitiveness of the electronic health record market and its implications for interoperability","volume":"136","author":"Sorace","year":"2020","journal-title":"Int J Med Inform"},{"issue":"7494","key":"2024040320013586400_ocae019-B2","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","year":"2005","journal-title":"BMJ"},{"issue":"3","key":"2024040320013586400_ocae019-B3","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1001\/archinternmed.2008.551","article-title":"Overrides of medication alerts in ambulatory care","volume":"169","author":"Isaac","year":"2009","journal-title":"Arch Intern Med"},{"issue":"3","key":"2024040320013586400_ocae019-B4","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1136\/amiajnl-2013-001813","article-title":"Overrides of medication-related clinical decision support alerts in outpatients","volume":"21","author":"Nanji","year":"2014","journal-title":"J Am Med Informatics Assoc"},{"key":"2024040320013586400_ocae019-B5","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.ijmedinf.2017.08.008","article-title":"Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients","volume":"107","author":"Zenziper Straichman","year":"2017","journal-title":"Int J Med Inform"},{"issue":"11","key":"2024040320013586400_ocae019-B6","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1177\/1060028019863419","article-title":"The effect of patient-specific drug-drug interaction alerting on the frequency of alerts: a pilot study","volume":"53","author":"Horn","year":"2019","journal-title":"Ann Pharmacother"},{"issue":"1","key":"2024040320013586400_ocae019-B7","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1186\/s12911-021-01465-2","article-title":"A systematic review of theoretical constructs in CDS literature","volume":"21","author":"Liu","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"2024040320013586400_ocae019-B8","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1197\/jamia.M2808","article-title":"Tiering drug-drug interaction alerts by severity increases compliance rates","volume":"16","author":"Paterno","year":"2009","journal-title":"J Am Med Informatics Assoc"},{"issue":"3","key":"2024040320013586400_ocae019-B9","doi-asserted-by":"crossref","DOI":"10.1542\/peds.2017-4111","article-title":"Optimizing drug-drug interaction alerts using a multidimensional approach","volume":"143","author":"Daniels","year":"2019","journal-title":"Pediatrics"},{"issue":"2","key":"2024040320013586400_ocae019-B10","doi-asserted-by":"crossref","first-page":"144","DOI":"10.2146\/ajhp140095","article-title":"Reduction of clinical support warnings through recategorization of severity levels","volume":"72","author":"Parke","year":"2015","journal-title":"Am J Health Syst Pharm"},{"key":"2024040320013586400_ocae019-B11","first-page":"845","article-title":"Medical alert management: a real-time adaptive decision support tool to reduce alert fatigue","volume":"2014","author":"Lee","year":"2014","journal-title":"AMIA Annu Symp Proc"},{"issue":"4","key":"2024040320013586400_ocae019-B12","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1197\/jamia.M2606","article-title":"A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care","volume":"15","author":"Tamblyn","year":"2008","journal-title":"J Am Med Informatics Assoc"},{"key":"2024040320013586400_ocae019-B13","author":"Clickbusters | Vanderbilt Clinical Informatics Center","year":"2021"},{"issue":"4","key":"2024040320013586400_ocae019-B14","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1002\/pds.3591","article-title":"Differences among admitting departments in alerts and alert overrides for drug-drug interaction","volume":"23","author":"Ahn","year":"2014","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"6","key":"2024040320013586400_ocae019-B15","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1093\/jamia\/ocv011","article-title":"Recommendations to improve the usability of drug-drug interaction clinical decision support alerts","volume":"22","author":"Payne","year":"2015","journal-title":"J Am Med Informatics Assoc"},{"issue":"1","key":"2024040320013586400_ocae019-B16","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1186\/s12911-017-0430-8","article-title":"Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system","volume":"17","author":"Ancker","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"issue":"11","key":"2024040320013586400_ocae019-B17","doi-asserted-by":"crossref","first-page":"2514","DOI":"10.1093\/jamia\/ocab160","article-title":"A theory-based meta-regression of factors influencing clinical decision support adoption and implementation","volume":"28","author":"Liu","year":"2021","journal-title":"J Am Med Informatics Assoc"},{"issue":"13","key":"2024040320013586400_ocae019-B18","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.1093\/ajhp\/zxac045","article-title":"Improving the specificity of drug-drug interaction alerts: can it be done?","volume":"79","author":"Reese","year":"2022","journal-title":"Am J Health Syst Pharm"},{"key":"2024040320013586400_ocae019-B19","author":"Arrieta","year":"2019"},{"key":"2024040320013586400_ocae019-B20","author":"Gunning","year":"2017"},{"key":"2024040320013586400_ocae019-B21","author":"Das","year":"2020"},{"key":"2024040320013586400_ocae019-B22","first-page":"97","author":"Ribeiro","year":"2016"},{"key":"2024040320013586400_ocae019-B23","author":"Ribeiro","year":"2018"},{"issue":"5","key":"2024040320013586400_ocae019-B24","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1093\/jamia\/ocab292","article-title":"The potential for leveraging machine learning to filter medication alerts","volume":"29","author":"Liu","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"1","key":"2024040320013586400_ocae019-B25","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1023\/A:1022631118932","article-title":"Very simple classification rules perform well on most commonly used datasets","volume":"11","author":"Holte","year":"1993","journal-title":"Mach Learn"},{"issue":"1","key":"2024040320013586400_ocae019-B26","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1093\/jamia\/ocac210","article-title":"New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record","volume":"30","author":"Liu","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"2","key":"2024040320013586400_ocae019-B27","doi-asserted-by":"crossref","first-page":"e0246306","DOI":"10.1371\/journal.pone.0246306","article-title":"Predicting mortality of patients with acute kidney injury in the ICU using XGBoost model","volume":"16","author":"Liu","year":"2021","journal-title":"PLoS One"},{"issue":"1","key":"2024040320013586400_ocae019-B28","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1186\/s40537-020-00305-w","article-title":"Survey on categorical data for neural networks","volume":"7","author":"Hancock","year":"2020","journal-title":"J Big Data"},{"key":"2024040320013586400_ocae019-B29","author":"Gautier","year":"2023"},{"issue":"1","key":"2024040320013586400_ocae019-B30","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A decision-theoretic generalization of on-line learning and an application to boosting","volume":"55","author":"Freund","year":"1997","journal-title":"J Comput Syst Sci"},{"issue":"61","key":"2024040320013586400_ocae019-B31","doi-asserted-by":"crossref","first-page":"3192","DOI":"10.21105\/joss.03192","article-title":"imodels: a python package for fitting interpretable models","volume":"6","author":"Singh","year":"2021","journal-title":"JOSS"},{"key":"2024040320013586400_ocae019-B32","author":"Nori","year":"2019"},{"key":"2024040320013586400_ocae019-B33","author":"Akiba"},{"issue":"6","key":"2024040320013586400_ocae019-B34","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1093\/jamia\/ocac027","article-title":"Clinician collaboration to improve clinical decision support: the Clickbusters initiative","volume":"29","author":"McCoy","year":"2022","journal-title":"J Am Med Informatics Assoc"},{"issue":"7","key":"2024040320013586400_ocae019-B35","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Informatics Assoc"},{"key":"2024040320013586400_ocae019-B36","first-page":"37","author":"Aaron"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/4\/968\/57148447\/ocae019.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/4\/968\/57148447\/ocae019.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T20:01:58Z","timestamp":1712174518000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/4\/968\/7610153"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,22]]},"references-count":36,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,2,22]]},"published-print":{"date-parts":[[2024,4,3]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae019","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,4,1]]},"published":{"date-parts":[[2024,2,22]]}}}