{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T16:05:48Z","timestamp":1777046748537,"version":"3.51.4"},"reference-count":50,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2018,5,11]],"date-time":"2018-05-11T00:00:00Z","timestamp":1525996800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/academic.oup.com\/journals\/pages\/about_us\/legal\/notices"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM011966"],"award-info":[{"award-number":["R01LM011966"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic\u00ae (Epic Systems Corporation, Madison, WI, USA) at Brigham and Women\u2019s Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Malfunctions\/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocy041","type":"journal-article","created":{"date-parts":[[2018,4,3]],"date-time":"2018-04-03T19:36:54Z","timestamp":1522784214000},"page":"862-871","source":"Crossref","is-referenced-by-count":37,"title":["Using statistical anomaly detection models to find clinical decision support malfunctions"],"prefix":"10.1093","volume":"25","author":[{"given":"Soumi","family":"Ray","sequence":"first","affiliation":[{"name":"Department of General Internal Medicine and Primary Care, Brigham & Women\u2019s Hospital, Boston, MA, USA"},{"name":"Department of Medicine, Harvard Medical School, Boston, MA, USA"}]},{"given":"Dustin S","family":"McEvoy","sequence":"additional","affiliation":[{"name":"Partners Healthcare, Information Systems, Somerville, MA, USA"}]},{"given":"Skye","family":"Aaron","sequence":"additional","affiliation":[{"name":"Department of General Internal Medicine and Primary Care, Brigham & Women\u2019s Hospital, Boston, MA, USA"}]},{"given":"Thu-Trang","family":"Hickman","sequence":"additional","affiliation":[{"name":"Department of General Internal Medicine and Primary Care, Brigham & Women\u2019s Hospital, Boston, MA, USA"}]},{"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of General Internal Medicine and Primary Care, Brigham & Women\u2019s Hospital, Boston, MA, USA"},{"name":"Department of Medicine, Harvard Medical School, Boston, MA, USA"},{"name":"Partners Healthcare, Information Systems, Somerville, MA, USA"}]}],"member":"286","published-online":{"date-parts":[[2018,5,11]]},"reference":[{"issue":"6","key":"2020110612381880000_ocy041-B1","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1056\/NEJMp1006114","article-title":"The \u201cmeaningful use\u201d regulation for electronic health records","volume":"363","author":"Blumenthal","year":"2010","journal-title":"N Engl J Med"},{"issue":"6","key":"2020110612381880000_ocy041-B2","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1111\/joim.12119","article-title":"Electronic health records: new opportunities for clinical research","volume":"274","author":"Coorevits","year":"2013","journal-title":"J Intern Med"},{"issue":"2","key":"2020110612381880000_ocy041-B3","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1197\/jamia.M2334","article-title":"A roadmap for national action on clinical decision support","volume":"14","author":"Osheroff","year":"2007","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"2020110612381880000_ocy041-B4","doi-asserted-by":"crossref","first-page":"746","DOI":"10.4338\/ACI-2014-02-RA-0018","article-title":"Reducing risk with clinical decision support","volume":"05","author":"Zuccotti","year":"2014","journal-title":"Appl Clin Inform"},{"issue":"7494","key":"2020110612381880000_ocy041-B5","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","year":"2005","journal-title":"BMJ"},{"issue":"6","key":"2020110612381880000_ocy041-B6","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1111\/j.1365-2753.2009.01201.x","article-title":"The intended and unintended consequences of clinical guidelines","volume":"15","author":"Shackelton","year":"2009","journal-title":"J Eval Clin Pract"},{"issue":"1","key":"2020110612381880000_ocy041-B7","doi-asserted-by":"crossref","first-page":"29","DOI":"10.7326\/0003-4819-157-1-201207030-00450","article-title":"Effect of clinical decision-support systems: a systematic review","volume":"157","author":"Bright","year":"2012","journal-title":"Ann Intern Med"},{"issue":"10","key":"2020110612381880000_ocy041-B8","doi-asserted-by":"crossref","first-page":"1223","DOI":"10.1001\/jama.293.10.1223","article-title":"Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review","volume":"293","author":"Garg","year":"2005","journal-title":". JAMA"},{"issue":"6","key":"2020110612381880000_ocy041-B9","doi-asserted-by":"crossref","first-page":"1068","DOI":"10.1093\/jamia\/ocw005","article-title":"Analysis of clinical decision support system malfunctions: a case series and survey","volume":"23","author":"Wright","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"2020110612381880000_ocy041-B10","doi-asserted-by":"crossref","first-page":"910","DOI":"10.4338\/ACI-2017-01-RA-0006","article-title":"Clinical decisions support malfunctions in a commercial electronic health record","volume":"08","author":"Kassakian","year":"2017","journal-title":"Appl Clin Inform"},{"issue":"e1","key":"2020110612381880000_ocy041-B11","doi-asserted-by":"crossref","first-page":"e187","DOI":"10.1136\/amiajnl-2012-001494","article-title":"The hazard of software updates to clinical workstations: a natural experiment","volume":"20","author":"Landman","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020110612381880000_ocy041-B12","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1136\/amiajnl-2011-000185","article-title":"A framework for evaluating the appropriateness of clinical decision support alerts and responses","volume":"19","author":"McCoy","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612381880000_ocy041-B13","article-title":"Clinical decision support alert malfunctions: analysis and empirically derived taxonomy","author":"Wright","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020110612381880000_ocy041-B14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1541880.1541882","article-title":"Anomaly detection","volume":"41","author":"Chandola","year":"2009","journal-title":"ACM Comput Surv"},{"key":"2020110612381880000_ocy041-B15","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-6396-2","volume-title":"Outlier Analysis","author":"Aggarwal","year":"2013"},{"issue":"2","key":"2020110612381880000_ocy041-B16","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1023\/B:AIRE.0000045502.10941.a9","article-title":"A survey of outlier detection methodologies","volume":"22","author":"Hodge","journal-title":"Artif Intell Rev"},{"issue":"1","key":"2020110612381880000_ocy041-B17","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1214\/aos\/1176343001","article-title":"On tests for detecting change in mean","volume":"3","author":"Sen","year":"1975","journal-title":"Ann Stat"},{"issue":"3","key":"2020110612381880000_ocy041-B18","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1093\/biomet\/57.3.477","article-title":"Inference about the change-point in a sequence of random variables","volume":"57","author":"Hinkley","year":"1970","journal-title":"Biometrika"},{"issue":"9","key":"2020110612381880000_ocy041-B19","doi-asserted-by":"crossref","first-page":"2250","DOI":"10.1109\/TKDE.2013.184","article-title":"Outlier detection for temporal data: a survey","volume":"26","author":"Gupta","year":"2014","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"2","key":"2020110612381880000_ocy041-B20","doi-asserted-by":"crossref","first-page":"126","DOI":"10.2307\/2346729","article-title":"A non-parametric approach to the change-point problem","volume":"28","author":"Pettitt","year":"1979","journal-title":"Appl Stat"},{"issue":"6","key":"2020110612381880000_ocy041-B21","doi-asserted-by":"crossref","first-page":"1696","DOI":"10.1016\/j.csda.2009.12.005","article-title":"Testing, monitoring, and dating structural changes in exchange rate regimes","volume":"54","author":"Zeileis","year":"2010","journal-title":"Comput Stat Data Anal"},{"issue":"24","key":"2020110612381880000_ocy041-B22","doi-asserted-by":"crossref","first-page":"7454","DOI":"10.1073\/pnas.1415846112","article-title":"Identifying localized changes in large systems: change-point detection for biomolecular simulations","volume":"112","author":"Fan","year":"2015","journal-title":"Proc Natl Acad Sci USA"},{"key":"2020110612381880000_ocy041-B23","article-title":"Bayesian online changepoint detection","author":"Adams"},{"issue":"04","key":"2020110612381880000_ocy041-B24","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1093\/pan\/mph023","article-title":"A Bayesian change point model for historical time series analysis","volume":"12","author":"Western","year":"2004","journal-title":"Polit Anal"},{"issue":"4","key":"2020110612381880000_ocy041-B25","first-page":"W00D28.","article-title":"Real-time Bayesian anomaly detection in streaming environmental data","volume":"46","author":"Hill","year":"2016","journal-title":"Water Resour Res"},{"issue":"3","key":"2020110612381880000_ocy041-B26","doi-asserted-by":"crossref","first-page":"948","DOI":"10.1016\/j.ijforecast.2015.06.001","article-title":"Probabilistic anomaly detection in natural gas time series data","volume":"32","author":"Akouemo","year":"2016","journal-title":"Int J Forecast"},{"key":"2020110612381880000_ocy041-B27","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.jbi.2016.10.002","article-title":"Outlier-based detection of unusual patient-management actions: an ICU study","volume":"64","author":"Hauskrecht","year":"2016","journal-title":"J Biomed Inform"},{"key":"2020110612381880000_ocy041-B28","first-page":"185","volume-title":"Detecting Anomalies in Alert Firing within Clinical Decision Support Systems Using Anomaly\/Outlier Detection Techniques","author":"Ray"},{"key":"2020110612381880000_ocy041-B29","author":"Vallis","year":"2014"},{"key":"2020110612381880000_ocy041-B30","doi-asserted-by":"crossref","article-title":"Leveraging cloud data to mitigate user experience from \u201cbreaking bad","author":"James","DOI":"10.1109\/BigData.2016.7841013"},{"issue":"2","key":"2020110612381880000_ocy041-B31","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1080\/00949658708810988","article-title":"On testing homogeneity of variances for gaussian models","volume":"27","author":"Jen","year":"1987","journal-title":"J Stat Comput Simul"},{"issue":"3","key":"2020110612381880000_ocy041-B32","doi-asserted-by":"crossref","first-page":"507","DOI":"10.2307\/2529204","article-title":"A cluster analysis method for grouping means in the analysis of variance","volume":"30","author":"Scott","year":"1974","journal-title":"Biometrics"},{"issue":"3","key":"2020110612381880000_ocy041-B33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v058.i03","article-title":"changepoint: an R package for changepoint analysis","volume":"58","author":"Killick","year":"2014","journal-title":"J Stat Soft"},{"key":"2020110612381880000_ocy041-B34","article-title":"Parametric Statistical Change Point Analysis","author":"Chen","year":"2000","journal-title":"Birkh\u00e4user"},{"key":"2020110612381880000_ocy041-B35","author":"L\u00f3pez-de-Lacalle","year":"2017"},{"key":"2020110612381880000_ocy041-B36","author":"Pena","year":"2013"},{"issue":"421","key":"2020110612381880000_ocy041-B37","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1080\/01621459.1993.10594321","article-title":"Joint estimation of model parameters and outlier effects in time series","volume":"88","author":"Chen","year":"1993","journal-title":"J Am Stat Assoc"},{"issue":"7","key":"2020110612381880000_ocy041-B38","first-page":"1","article-title":"ecp: an r package for nonparametric multiple change point analysis of multivariate data","volume":"62","author":"Matteson","year":"2015","journal-title":"J. Stat. Softw"},{"issue":"2","key":"2020110612381880000_ocy041-B39","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1007\/s00357-005-0012-9","article-title":"Hierarchical clustering via joint between-within distances: extending ward\u2019s minimum variance method","volume":"22","author":"Szekely","year":"2005","journal-title":"J Classif"},{"key":"2020110612381880000_ocy041-B40","first-page":"55","article-title":"Detecting disorder in multidimensional random process","volume":"24","author":"Vostrikova","year":"1981","journal-title":"Soviet Math Dokl"},{"issue":"488","key":"2020110612381880000_ocy041-B41","doi-asserted-by":"crossref","first-page":"1504","DOI":"10.1198\/jasa.2009.tm08368","article-title":"Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk","volume":"104","author":"Gandy","year":"2009","journal-title":"J Am Stati Assoc"},{"issue":"3","key":"2020110612381880000_ocy041-B42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v023.i03","article-title":"bcp: an RPackage for performing a Bayesian analysis of change point problems","volume":"23","author":"Erdman","year":"2007","journal-title":"J Stat Soft"},{"issue":"421","key":"2020110612381880000_ocy041-B43","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1080\/01621459.1993.10594323","article-title":"A Bayesian analysis for change point problems","volume":"88","author":"Barry","year":"1993","journal-title":"J Am Stat Assoc"},{"key":"2020110612381880000_ocy041-B44","first-page":"3","article-title":"STL: a seasonal-trend decomposition procedure based on loess","volume":"6","author":"Cleveland","year":"1990","journal-title":"J Off Stat"},{"issue":"2","key":"2020110612381880000_ocy041-B45","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1080\/00401706.1983.10487848","article-title":"Percentage points for a generalized ESD many-outlier procedure","volume":"25","author":"Rosner","year":"1983","journal-title":"Technometrics"},{"issue":"2","key":"2020110612381880000_ocy041-B46","doi-asserted-by":"crossref","first-page":"221","DOI":"10.2307\/1268354","article-title":"On the detection of many outliers","volume":"17","author":"Rosner","year":"1975","journal-title":"Technometrics"},{"key":"2020110612381880000_ocy041-B47","first-page":"157","volume-title":"The Notion of Breakdown Point. A Festschrift for Erich L. Lehmann","author":"Donoho","year":"1983"},{"issue":"2","key":"2020110612381880000_ocy041-B48","doi-asserted-by":"crossref","first-page":"240","DOI":"10.1080\/00401706.1987.10488218","article-title":"Robust statistics: the approach based on influence functions","volume":"29","author":"Ruppert","year":"1987","journal-title":"Technometrics"},{"issue":"346","key":"2020110612381880000_ocy041-B49","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1080\/01621459.1974.10482962","article-title":"The influence curve and its role in robust estimation","volume":"69","author":"Hampel","year":"1974","journal-title":"J Am Stat Assoc"},{"key":"2020110612381880000_ocy041-B50","volume-title":"Permutation, Parametric and Bootstrap Tests of Hypotheses","author":"Good","year":"2005"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/7\/862\/34150697\/ocy041.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/7\/862\/34150697\/ocy041.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,5]],"date-time":"2024-07-05T18:09:28Z","timestamp":1720202968000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/7\/862\/4995314"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5,11]]},"references-count":50,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2018,5,11]]},"published-print":{"date-parts":[[2018,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy041","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,7]]},"published":{"date-parts":[[2018,5,11]]}}}