{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T22:36:42Z","timestamp":1769639802498,"version":"3.49.0"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2022,1,5]],"date-time":"2022-01-05T00:00:00Z","timestamp":1641340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100007747","name":"University of Utah","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007747","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user\u2019s view.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate &amp;lt;0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1\u2014there is no difference in precision among prediction models; and H2\u2014the removal of any feature category does not change precision.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P\u2009&amp;lt;\u20090.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (\u22120.147, P\u2009=\u20090.001).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Machine learning potentially enables the intelligent filtering of medication alerts.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab292","type":"journal-article","created":{"date-parts":[[2021,12,24]],"date-time":"2021-12-24T12:08:29Z","timestamp":1640347709000},"page":"891-899","source":"Crossref","is-referenced-by-count":22,"title":["The potential for leveraging machine learning to filter medication alerts"],"prefix":"10.1093","volume":"29","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4282-9338","authenticated-orcid":false,"given":"Kensaku","family":"Kawamoto","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9954-6799","authenticated-orcid":false,"given":"Guilherme","family":"Del Fiol","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8297-2860","authenticated-orcid":false,"given":"Charlene","family":"Weir","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel C","family":"Malone","sequence":"additional","affiliation":[{"name":"Department of Pharmacotherapy, Skaggs College of Pharmacy, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1081-1670","authenticated-orcid":false,"given":"Thomas J","family":"Reese","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"},{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9140-4454","authenticated-orcid":false,"given":"Keaton","family":"Morgan","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"ElHalta","sequence":"additional","affiliation":[{"name":"Pharmacy Services, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6396-3259","authenticated-orcid":false,"given":"Samir","family":"Abdelrahman","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"},{"name":"Computer Science Department, Faculty of Computers and Artificial Intelligence, Cairo University, Giza, Egypt"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2022,1,5]]},"reference":[{"issue":"2","key":"2022041311574765900_ocab292-B1","doi-asserted-by":"crossref","first-page":"491","DOI":"10.4338\/ACI-2016-10-RA-0168","article-title":"The effects of medication alerts on prescriber response in a pediatric hospital","volume":"8","author":"Dexheimer","year":"2017","journal-title":"Appl Clin Inform"},{"issue":"23","key":"2022041311574765900_ocab292-B2","doi-asserted-by":"crossref","first-page":"2098","DOI":"10.2146\/ajhp090181","article-title":"Alert fatigue","volume":"66","author":"Cash","year":"2009","journal-title":"Am J Health Syst Pharm"},{"issue":"2","key":"2022041311574765900_ocab292-B3","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1197\/jamia.M2334","article-title":"A roadmap for national action on clinical decision support","volume":"14","author":"Osheroff","year":"2007","journal-title":"J Am Med Inform Assoc"},{"issue":"7494","key":"2022041311574765900_ocab292-B4","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","year":"2005","journal-title":"BMJ"},{"issue":"1","key":"2022041311574765900_ocab292-B5","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1055\/s-0039-3402757","article-title":"Reducing interruptive alert burden using quality improvement methodology","volume":"11","author":"Chaparro","year":"2020","journal-title":"Appl Clin Inform"},{"issue":"2","key":"2022041311574765900_ocab292-B6","first-page":"195","article-title":"Clinical decision support alert appropriateness: a review and proposal for improvement","volume":"14","author":"McCoy","year":"2014","journal-title":"Ochsner J"},{"issue":"9","key":"2022041311574765900_ocab292-B7","doi-asserted-by":"crossref","first-page":"1481","DOI":"10.1097\/CCM.0000000000002580","article-title":"Technologic distractions (part 1): summary of approaches to manage alert quantity with intent to reduce alert fatigue and suggestions for alert fatigue metrics","volume":"45","author":"Kane-Gill","year":"2017","journal-title":"Crit Care Med"},{"issue":"1","key":"2022041311574765900_ocab292-B8","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1186\/s12911-021-01465-2","article-title":"A systematic review of theoretical constructs in CDS literature","volume":"21","author":"Liu","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"2022041311574765900_ocab292-B9","author":"Liu","year":"2020"},{"key":"2022041311574765900_ocab292-B10","first-page":"2514","article-title":"A theory-based meta-regression of factors influencing clinical decision support adoption and implementation","author":"Liu","journal-title":"J Am Med Informatics Assoc"},{"issue":"1","key":"2022041311574765900_ocab292-B11","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1197\/jamia.M2808","article-title":"Tiering drug-drug interaction alerts by severity increases compliance rates","volume":"16","author":"Paterno","year":"2009","journal-title":"J Am Med Informatics Assoc"},{"key":"2022041311574765900_ocab292-B12","first-page":"624","article-title":"A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study","volume":"2018","author":"Kawamanto","year":"2018","journal-title":"AMIA. Annu Symp Proceedings AMIA Symp"},{"issue":"6","key":"2022041311574765900_ocab292-B13","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1093\/jamia\/ocv011","article-title":"Recommendations to improve the usability of drug-drug interaction clinical decision support alerts","volume":"22","author":"Payne","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2022041311574765900_ocab292-B14","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1186\/s12911-017-0430-8","article-title":"Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system","volume":"17","author":"Ancker","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"key":"2022041311574765900_ocab292-B15","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1186\/1472-6947-12-111","article-title":"Physicians\u2019 perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems","volume":"12","author":"Jung","year":"2012","journal-title":"BMC Med Inform Decis Mak"},{"key":"2022041311574765900_ocab292-B16","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1186\/1472-6947-11-35","article-title":"Development of a context model to prioritize drug safety alerts in CPOE systems","volume":"11","author":"Riedmann","year":"2011","journal-title":"BMC Med Inform Decis Mak"},{"key":"2022041311574765900_ocab292-B17","first-page":"920","article-title":"Contextualization of automatic alerts during electronic prescription: researchers\u2019 and users\u2019 opinions on useful context factors","volume":"169","author":"Ammenwerth","year":"2011","journal-title":"Stud Health Technol Inform"},{"issue":"6","key":"2022041311574765900_ocab292-B18","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1136\/amiajnl-2010-000006","article-title":"How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study","volume":"18","author":"Riedmann","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2022041311574765900_ocab292-B19","doi-asserted-by":"crossref","first-page":"e19489","DOI":"10.2196\/19489","article-title":"Machine learning approach to reduce alert fatigue using a disease medication\u2013related clinical decision support system: model development and validation","volume":"8","author":"Poly","year":"2020","journal-title":"JMIR Med Inform"},{"key":"2022041311574765900_ocab292-B20","first-page":"624","article-title":"A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study","volume":"2018","author":"Kawamanto","year":"2018","journal-title":"AMIA Annu Symp Proceedings AMIA Symp"},{"issue":"8","key":"2022041311574765900_ocab292-B21","doi-asserted-by":"crossref","first-page":"895","DOI":"10.1093\/oxfordjournals.aje.a116164","article-title":"Biased estimation of the odds ratio in case-control studies due to the use of ad hoc methods of correcting for missing values for confounding variables","volume":"134","author":"Vach","year":"1991","journal-title":"Am J Epidemiol"},{"key":"2022041311574765900_ocab292-B22","volume-title":"Missing Data in Randomised Controlled Trials\u2014A Practical Guide","author":"Carpenter","year":"2008"},{"issue":"4","key":"2022041311574765900_ocab292-B23","doi-asserted-by":"crossref","first-page":"7","DOI":"10.5120\/ijca2017915495","article-title":"A comparative study of categorical variable encoding techniques for neural network classifiers","volume":"175","author":"Potdar","year":"2017","journal-title":"IJCA"},{"issue":"10","key":"2022041311574765900_ocab292-B24","doi-asserted-by":"crossref","first-page":"1104","DOI":"10.1016\/j.compbiomed.2005.09.002","article-title":"Machine learning in bioinformatics: a brief survey and recommendations for practitioners","volume":"36","author":"Bhaskar","year":"2006","journal-title":"Comput Biol Med"},{"key":"2022041311574765900_ocab292-B25","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"JAIR"},{"issue":"260","key":"2022041311574765900_ocab292-B26","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","article-title":"Use of ranks in one-criterion variance analysis","volume":"47","author":"Kruskal","year":"1952","journal-title":"J Am Stat Assoc"},{"key":"2022041311574765900_ocab292-B27","first-page":"11","volume-title":"Breakthroughs in Statistics. Springer Series in Statistics (Perspectives in Statistics)","author":"Pearson","year":"1992"},{"key":"2022041311574765900_ocab292-B28","first-page":"1","volume-title":"Encyclopedia of Public Health","author":"Benesty","year":"2009"},{"key":"2022041311574765900_ocab292-B29","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/978-1-4614-6849-3_3","volume-title":"Applied Predictive Modeling","author":"Kuhn","year":"2013"},{"key":"2022041311574765900_ocab292-B30","doi-asserted-by":"crossref","DOI":"10.1002\/0471722146","volume-title":"Applied Logistic Regression","author":"Hosmer","year":"2000"},{"issue":"1","key":"2022041311574765900_ocab292-B31","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach Learn"},{"issue":"10","key":"2022041311574765900_ocab292-B32","doi-asserted-by":"crossref","first-page":"993","DOI":"10.1109\/34.58871","article-title":"Neural network ensembles","volume":"12","author":"Hansen","year":"1990","journal-title":"IEEE Trans Pattern Anal Machine Intell"},{"key":"2022041311574765900_ocab292-B33","author":"Dorogush","year":"2018"},{"key":"2022041311574765900_ocab292-B34","first-page":"3147","article-title":"LightGBM: a highly efficient gradient boosting decision tree","author":"Ke","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"3","key":"2022041311574765900_ocab292-B35","doi-asserted-by":"crossref","first-page":"e190968","DOI":"10.1001\/jamanetworkopen.2019.0968","article-title":"Evaluation of machine-learning algorithms for predicting opioid overdose risk among Medicare beneficiaries with opioid prescriptions","volume":"2","author":"Lo-Ciganic","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2022041311574765900_ocab292-B36","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.cmpb.2018.03.021","article-title":"Supervised signal detection for adverse drug reactions in medication dispensing data","volume":"161","author":"Hoang","year":"2018","journal-title":"Comput Methods Programs Biomed"},{"issue":"2","key":"2022041311574765900_ocab292-B37","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1136\/amiajnl-2013-001854","article-title":"Medical decision support using machine learning for early detection of late-onset neonatal sepsis","volume":"21","author":"Mani","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2022041311574765900_ocab292-B38","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","author":"Demsar","year":"2006","journal-title":"J Mach Learn Res"},{"key":"2022041311574765900_ocab292-B39","year":"2017"},{"key":"2022041311574765900_ocab292-B40","year":"2020"},{"issue":"2","key":"2022041311574765900_ocab292-B41","doi-asserted-by":"crossref","first-page":"e0246306","DOI":"10.1371\/journal.pone.0246306","article-title":"Predicting mortality of patients with acute kidney injury in the ICU using XGBoost model","volume":"16","author":"Liu","year":"2021","journal-title":"PLoS One"},{"issue":"19","key":"2022041311574765900_ocab292-B42","doi-asserted-by":"crossref","first-page":"e25813","DOI":"10.1097\/MD.0000000000025813","article-title":"Predicting in-hospital mortality in ICU patients with sepsis using gradient boosting decision tree","volume":"100","author":"Li","year":"2021","journal-title":"Medicine (Baltimore)"},{"issue":"4","key":"2022041311574765900_ocab292-B43","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1016\/j.ijmedinf.2013.12.006","article-title":"What, if all alerts were specific\u2014estimating the potential impact on drug interaction alert burden","volume":"83","author":"Seidling","year":"2014","journal-title":"Int J Med Inform"},{"issue":"4","key":"2022041311574765900_ocab292-B44","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1136\/amiajnl-2010-000039","article-title":"Factors influencing alert acceptance: A novel approach for predicting the success of clinical decision support","volume":"18","author":"Seidling","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2022041311574765900_ocab292-B45","doi-asserted-by":"crossref","first-page":"e20174111","DOI":"10.1542\/peds.2017-4111","article-title":"Optimizing drug-drug interaction alerts using a multidimensional approach","volume":"143","author":"Daniels","year":"2019","journal-title":"Pediatrics"},{"issue":"8","key":"2022041311574765900_ocab292-B46","doi-asserted-by":"crossref","first-page":"662","DOI":"10.2146\/ajhp100465","article-title":"Customizing clinical decision support to prevent excessive drug\u2013drug interaction alerts","volume":"68","author":"Horn","year":"2011","journal-title":"Am J Health Syst Pharm"},{"issue":"11","key":"2022041311574765900_ocab292-B47","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1093\/jamia\/ocaa154","article-title":"A machine learning\u2013based clinical decision support system to identify prescriptions with a high risk of medication error","volume":"27","author":"Corny","year":"2020","journal-title":"J Am Med Inform Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/891\/43372317\/ocab292.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/5\/891\/43372317\/ocab292.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T12:11:49Z","timestamp":1649851909000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/5\/891\/6497966"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,5]]},"references-count":47,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,1,5]]},"published-print":{"date-parts":[[2022,4,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab292","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5,1]]},"published":{"date-parts":[[2022,1,5]]}}}