{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T01:39:55Z","timestamp":1772156395099,"version":"3.50.1"},"reference-count":29,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2018,12,25]],"date-time":"2018-12-25T00:00:00Z","timestamp":1545696000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM011966"],"award-info":[{"award-number":["R01LM011966"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Rule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.<\/jats:p><\/jats:sec><jats:sec><jats:title>Objective<\/jats:title><jats:p>Investigate whether user override comments can be used to discover malfunctions.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: \u201cbroken,\u201d \u201cnot broken, but could be improved,\u201d and \u201cnot broken.\u201d We used 3 methods (frequency of comments, cranky word list heuristic, and a Na\u00efve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Of the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC\u2009=\u20090.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC\u2009=\u20090.723. The Na\u00efve Bayes classifier had precision at 20 of 17 and AUC\u2009=\u20090.738.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Override comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Override comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocy139","type":"journal-article","created":{"date-parts":[[2018,10,8]],"date-time":"2018-10-08T20:02:25Z","timestamp":1539028945000},"page":"37-43","source":"Crossref","is-referenced-by-count":34,"title":["Cranky comments: detecting clinical decision support malfunctions through free-text override reasons"],"prefix":"10.1093","volume":"26","author":[{"given":"Skye","family":"Aaron","sequence":"first","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital, Boston, Massachusetts, USA"}]},{"given":"Dustin S","family":"McEvoy","sequence":"additional","affiliation":[{"name":"Partners eCare, Partners HealthCare, Boston, Massachusetts, USA"}]},{"given":"Soumi","family":"Ray","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital, Boston, Massachusetts, USA"},{"name":"Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA"}]},{"given":"Thu-Trang T","family":"Hickman","sequence":"additional","affiliation":[{"name":"Partners Community Health, Partners HealthCare, Boston, Massachusetts, USA"}]},{"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital, Boston, Massachusetts, USA"},{"name":"Partners eCare, Partners HealthCare, Boston, Massachusetts, USA"},{"name":"Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA"},{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA"}]}],"member":"286","published-online":{"date-parts":[[2018,12,26]]},"reference":[{"issue":"18","key":"2020110613012702800_ocy139-B1","doi-asserted-by":"crossref","first-page":"2741","DOI":"10.1001\/archinte.160.18.2741","article-title":"Effects of computerized physician order entry on prescribing practices","volume":"160","author":"Teich","year":"2000","journal-title":"Arch Intern Med"},{"issue":"4","key":"2020110613012702800_ocy139-B2","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1056\/NEJM199801223380406","article-title":"A computer-assisted management program for antibiotics and other antiinfective agents","volume":"338","author":"Evans","year":"1998","journal-title":"N Engl J Med"},{"key":"2020110613012702800_ocy139-B3","author":"Byrne"},{"issue":"3","key":"2020110613012702800_ocy139-B4","doi-asserted-by":"crossref","first-page":"11","DOI":"10.7812\/TPP\/14-227","article-title":"a metrics taxonomy and reporting strategy for rule-based alerts","volume":"19","author":"Krall","year":"2015","journal-title":"Perm J"},{"issue":"2","key":"2020110613012702800_ocy139-B5","first-page":"195","article-title":"Clinical decision support alert appropriateness: a review and proposal for improvement","volume":"14","author":"McCoy","year":"2014","journal-title":"Ochsner J"},{"key":"2020110613012702800_ocy139-B6","first-page":"1","article-title":"Adoption of electronic health record systems among U.S. non-federal acute care hospitals: 2008\u20132014","volume":"23","author":"Charles","year":"2015","journal-title":"ONC Data Brief"},{"key":"2020110613012702800_ocy139-B7","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.ijmedinf.2017.05.011","article-title":"A systematic review of the effectiveness of interruptive medication prescribing alerts in hospital CPOE systems to change prescriber behavior and improve patient safety","volume":"105","author":"Page","year":"2017","journal-title":"Int J Med Inform"},{"issue":"6","key":"2020110613012702800_ocy139-B8","doi-asserted-by":"crossref","first-page":"1109","DOI":"10.1136\/amiajnl-2014-002777","article-title":"Don\u2019t fuel the fire: decreasing intravenous haloperidol use in high risk patients via a customized electronic alert","volume":"21","author":"Pell","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"13","key":"2020110613012702800_ocy139-B9","doi-asserted-by":"crossref","first-page":"1511","DOI":"10.1001\/archinte.1994.00420130107014","article-title":"Effect of computer-based alerts on the treatment and outcomes of hospitalized patients","volume":"154","author":"Rind","year":"1994","journal-title":"Arch Intern Med"},{"issue":"6","key":"2020110613012702800_ocy139-B10","doi-asserted-by":"crossref","first-page":"1068","DOI":"10.1093\/jamia\/ocw005","article-title":"Analysis of clinical decision support system malfunctions: a case series and survey","volume":"23","author":"Wright","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"2020110613012702800_ocy139-B11","doi-asserted-by":"crossref","first-page":"910","DOI":"10.4338\/ACI-2017-01-RA-0006","article-title":"Clinical decisions support malfunctions in a commercial electronic health record","volume":"08","author":"Kassakian","year":"2017","journal-title":"Appl Clin Inform"},{"issue":"1","key":"2020110613012702800_ocy139-B12","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1197\/jamia.M1868","article-title":"Improving acceptance of computerized prescribing alerts in ambulatory care","volume":"13","author":"Shah","year":"2006","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613012702800_ocy139-B13","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1093\/jamia\/ocx106","article-title":"Clinical decision support alert malfunctions: analysis and empirically derived taxonomy","volume":"25","author":"Wright","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020110613012702800_ocy139-B14","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1136\/amiajnl-2011-000185","article-title":"A framework for evaluating the appropriateness of clinical decision support alerts and responses","volume":"19","author":"McCoy","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2020110613012702800_ocy139-B15","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1197\/jamia.M2311","article-title":"Turning off frequently overridden drug alerts: limited opportunities for doing it safely","volume":"15","author":"van der Sijs","year":"2008","journal-title":"J Am Med Inform Assoc"},{"issue":"9","key":"2020110613012702800_ocy139-B16","doi-asserted-by":"crossref","first-page":"919","DOI":"10.1007\/s00228-009-0654-3","article-title":"Unintended consequences of reducing QT-alert overload in a computerized physician order entry system","volume":"65","author":"van der Sijs","year":"2009","journal-title":"Eur J Clin Pharmacol"},{"key":"2020110613012702800_ocy139-B17","first-page":"111","article-title":"Alert override reasons: a failure to communicate","author":"Chused","year":"2008 6","journal-title":"AMIA Annu Symp Proc"},{"issue":"4","key":"2020110613012702800_ocy139-B18","doi-asserted-by":"crossref","first-page":"280","DOI":"10.4258\/hir.2014.20.4.280","article-title":"Differences of reasons for alert overrides on contraindicated co-prescriptions by admitting department","volume":"20","author":"Ahn","year":"2014","journal-title":"Healthc Inform Res"},{"issue":"11","key":"2020110613012702800_ocy139-B19","doi-asserted-by":"crossref","first-page":"792","DOI":"10.1016\/j.ijmedinf.2010.08.003","article-title":"Coded entry versus free-text and alert overrides: what you get depends on how you ask","volume":"79","author":"Seidling","year":"2010","journal-title":"Int J Med Inform"},{"issue":"3","key":"2020110613012702800_ocy139-B20","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1136\/amiajnl-2013-001813","article-title":"Overrides of medication-related clinical decision support alerts in outpatients","volume":"21","author":"Nanji","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613012702800_ocy139-B21","author":"Kouloumpis"},{"key":"2020110613012702800_ocy139-B22","author":"Bollen"},{"issue":"6","key":"2020110613012702800_ocy139-B23","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1177\/0165551510388123","article-title":"Aspect-based sentiment analysis of movie reviews on discussion boards","volume":"36","author":"Thet","year":"2010","journal-title":"J Inf Sci"},{"key":"2020110613012702800_ocy139-B24","article-title":"A system for real-time Twitter sentiment analysis of 2012 U.S. presidential election cycle","volume-title":"ACL 2012 System Demonstrations","author":"Wang","year":"2012"},{"key":"2020110613012702800_ocy139-B25","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009"},{"key":"2020110613012702800_ocy139-B26","author":"Python Software Foundation"},{"key":"2020110613012702800_ocy139-B27","volume-title":"R: A Language and Environment for Statistical Computing","author":"R Core Team","year":"2017"},{"key":"2020110613012702800_ocy139-B28","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1186\/1471-2105-12-77","article-title":"pROC: an open-source package for R and S+ to analyze and compare ROC curves","volume":"12","author":"Robin","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2020110613012702800_ocy139-B29","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-98141-3","volume-title":"ggplot2: Elegant Graphics for Data Analysis","author":"Wickham","year":"2009"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/1\/37\/34151205\/ocy139.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/1\/37\/34151205\/ocy139.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,3]],"date-time":"2022-09-03T16:17:16Z","timestamp":1662221836000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/1\/37\/5260827"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12,26]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,12,26]]},"published-print":{"date-parts":[[2019,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy139","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,1]]},"published":{"date-parts":[[2018,12,26]]}}}