{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T17:38:56Z","timestamp":1771004336168,"version":"3.50.1"},"reference-count":36,"publisher":"Georg Thieme Verlag KG","issue":"05","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2022,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Objectives\u2003To improve clinical decision support (CDS) by allowing users to provide real-time feedback when they interact with CDS tools and by creating processes for responding to and acting on this feedback.<\/jats:p><jats:p>\n          Methods\u2003Two organizations implemented similar real-time feedback tools and processes in their electronic health record and gathered data over a 30-month period. At both sites, users could provide feedback by using Likert feedback links embedded in all end-user facing alerts, with results stored outside the electronic health record, and provide feedback as a comment when they overrode an alert. Both systems are monitored daily by clinical informatics teams.<\/jats:p><jats:p>\n          Results\u2003The two sites received 2,639 Likert feedback comments and 623,270 override comments over a 30-month period. Through four case studies, we describe our use of end-user feedback to rapidly respond to build errors, as well as identifying inaccurate knowledge management, user-interface issues, and unique workflows.<\/jats:p><jats:p>\n          Conclusion\u2003Feedback on CDS tools can be solicited in multiple ways, and it contains valuable and actionable suggestions to improve CDS alerts. Additionally, end users appreciate knowing their feedback is being received and may also make other suggestions to improve the electronic health record. Incorporation of end-user feedback into CDS monitoring, evaluation, and remediation is a way to improve CDS.<\/jats:p>","DOI":"10.1055\/s-0042-1757923","type":"journal-article","created":{"date-parts":[[2022,10,26]],"date-time":"2022-10-26T23:05:53Z","timestamp":1666825553000},"page":"1024-1032","source":"Crossref","is-referenced-by-count":15,"title":["Real-Time User Feedback to Support Clinical Decision Support System Improvement"],"prefix":"10.1055","volume":"13","author":[{"given":"David","family":"Rubins","sequence":"additional","affiliation":[{"name":"Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States"},{"name":"Digital, Mass General Brigham, Boston, Massachusetts, United States"}]},{"given":"Allison B.","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States"}]},{"given":"Sayon","family":"Dutta","sequence":"additional","affiliation":[{"name":"Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States"},{"name":"Digital, Mass General Brigham, Boston, Massachusetts, United States"},{"name":"Department of Emergency Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, Massachusetts, United States"}]},{"given":"Dustin S.","family":"McEvoy","sequence":"additional","affiliation":[{"name":"Digital, Mass General Brigham, Boston, Massachusetts, United States"}]},{"given":"Lorraine","family":"Patterson","sequence":"additional","affiliation":[{"name":"HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States"}]},{"given":"Amy","family":"Miller","sequence":"additional","affiliation":[{"name":"Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States"},{"name":"Digital, Mass General Brigham, Boston, Massachusetts, United States"}]},{"given":"John G.","family":"Jackson","sequence":"additional","affiliation":[{"name":"HealthIT, Vanderbilt University Medical Center, Nashville, Tennessee, United States"}]},{"given":"Gianna","family":"Zuccotti","sequence":"additional","affiliation":[{"name":"Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, Boston, Massachusetts, United States"},{"name":"Digital, Mass General Brigham, Boston, Massachusetts, United States"}]},{"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, United States"}]}],"member":"194","published-online":{"date-parts":[[2022,10,26]]},"reference":[{"issue":"10","key":"ref1","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1001\/archinte.166.10.1098","article-title":"The impact of prescribing safety alerts for elderly persons in an electronic medical record: an interrupted time series evaluation","volume":"166","author":"D H Smith","year":"2006","journal-title":"Arch Intern Med"},{"issue":"01","key":"ref2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.7326\/0003-4819-157-1-201207030-00450","article-title":"Effect of clinical decision-support systems: a systematic review","volume":"157","author":"T J Bright","year":"2012","journal-title":"Ann Intern Med"},{"issue":"12","key":"ref3","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1001\/archinte.163.12.1409","article-title":"Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review","volume":"163","author":"R Kaushal","year":"2003","journal-title":"Arch Intern Med"},{"issue":"17","key":"ref4","doi-asserted-by":"crossref","first-page":"1584","DOI":"10.1001\/jamainternmed.2013.8980","article-title":"Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial","volume":"173","author":"T G McGinn","year":"2013","journal-title":"JAMA Intern Med"},{"issue":"10","key":"ref5","doi-asserted-by":"crossref","first-page":"742","DOI":"10.7326\/0003-4819-144-10-200605160-00125","article-title":"Systematic review: impact of health information technology on quality, efficiency, and costs of medical care","volume":"144","author":"B Chaudhry","year":"2006","journal-title":"Ann Intern Med"},{"issue":"11","key":"ref6","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1016\/j.ijmedinf.2015.08.008","article-title":"Lessons learned from implementing service-oriented clinical decision support at four sites: a qualitative study","volume":"84","author":"A Wright","year":"2015","journal-title":"Int J Med Inform"},{"issue":"04","key":"ref7","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1136\/jamia.2001.0080299","article-title":"Reducing the frequency of errors in medicine using information technology","volume":"8","author":"D W Bates","year":"2001","journal-title":"J Am Med Inform Assoc"},{"issue":"25","key":"ref8","doi-asserted-by":"crossref","first-page":"2526","DOI":"10.1056\/NEJMsa020847","article-title":"Improving safety with information technology","volume":"348","author":"D W Bates","year":"2003","journal-title":"N Engl J Med"},{"key":"ref9","doi-asserted-by":"crossref","first-page":"m3216","DOI":"10.1136\/bmj.m3216","article-title":"Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials","volume":"370","author":"J L Kwan","year":"2020","journal-title":"BMJ"},{"issue":"05","key":"ref10","doi-asserted-by":"crossref","first-page":"1101","DOI":"10.1055\/s-0041-1739516","article-title":"Effect of electronic health record reminders for routine immunizations and immunizations needed for chronic medical conditions","volume":"12","author":"A B Stephens","year":"2021","journal-title":"Appl Clin Inform"},{"issue":"01","key":"ref11","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1055\/s-0041-1740483","article-title":"Effect of replacing vendor QTc alerts with a custom QTc risk alert in inpatients","volume":"13","author":"S Stettner","year":"2022","journal-title":"Appl Clin Inform"},{"issue":"05","key":"ref12","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1093\/jamia\/ocx115","article-title":"Medication-related clinical decision support alert overrides in inpatients","volume":"25","author":"K C Nanji","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"05","key":"ref13","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1093\/jamia\/ocx096","article-title":"Unintended adverse consequences of a clinical decision support system: two cases","volume":"25","author":"E G Stone","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"ref14","first-page":"26","article-title":"Some unintended consequences of clinical decision support systems","author":"J S Ash","year":"2007","journal-title":"AMIA Annu Symp Proc"},{"issue":"05","key":"ref15","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1093\/jamia\/ocx106","article-title":"Clinical decision support alert malfunctions: analysis and empirically derived taxonomy","volume":"25","author":"A Wright","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"ref16","doi-asserted-by":"crossref","first-page":"910","DOI":"10.4338\/ACI-2017-01-RA-0006","article-title":"Clinical decisions support malfunctions in a commercial electronic health record","volume":"8","author":"S Z Kassakian","year":"2017","journal-title":"Appl Clin Inform"},{"key":"ref17","doi-asserted-by":"crossref","first-page":"103453","DOI":"10.1016\/j.jbi.2020.103453","article-title":"Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow","volume":"106","author":"O O Olakotan","year":"2020","journal-title":"J Biomed Inform"},{"issue":"06","key":"ref18","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1197\/jamia.M1370","article-title":"Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality","volume":"10","author":"D W Bates","year":"2003","journal-title":"J Am Med Inform Assoc"},{"issue":"06","key":"ref19","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1093\/jamia\/ocv011","article-title":"Recommendations to improve the usability of drug-drug interaction clinical decision support alerts","volume":"22","author":"T H Payne","year":"2015","journal-title":"J Am Med Inform Assoc"},{"key":"ref20","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1186\/1748-5908-5-96","article-title":"Usability evaluation of a clinical decision support tool for osteoporosis disease management","volume":"5","author":"M Kastner","year":"2010","journal-title":"Implement Sci"},{"issue":"06","key":"ref21","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1016\/j.jbi.2012.09.002","article-title":"Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions","volume":"45","author":"J Horsky","year":"2012","journal-title":"J Biomed Inform"},{"key":"ref22","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.ijmedinf.2018.08.001","article-title":"Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: results of a Delphi study","volume":"118","author":"A Wright","year":"2018","journal-title":"Int J Med Inform"},{"issue":"01","key":"ref23","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1055\/s-0038-1632397","article-title":"The value of monitoring clinical decision support interventions","volume":"9","author":"E Yoshida","year":"2018","journal-title":"Appl Clin Inform"},{"issue":"02","key":"ref24","doi-asserted-by":"crossref","first-page":"e12471","DOI":"10.2196\/12471","article-title":"Live usability testing of two complex clinical decision support tools: observational study","volume":"6","author":"S Richardson","year":"2019","journal-title":"JMIR Human Factors"},{"issue":"06","key":"ref25","doi-asserted-by":"crossref","first-page":"1068","DOI":"10.1093\/jamia\/ocw005","article-title":"Analysis of clinical decision support system malfunctions: a case series and survey","volume":"23","author":"A Wright","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"15","key":"ref26","doi-asserted-by":"crossref","first-page":"1130","DOI":"10.2146\/ajhp160819","article-title":"Monitoring clinical decision support in the electronic health record","volume":"74","author":"J H Lam","year":"2017","journal-title":"Am J Health Syst Pharm"},{"key":"ref27","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/j.artmed.2018.06.003","article-title":"Change-point detection method for clinical decision support system rule monitoring","volume":"91","author":"S Liu","year":"2018","journal-title":"Artif Intell Med"},{"issue":"01","key":"ref28","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1093\/jamia\/ocy139","article-title":"Cranky comments: detecting clinical decision support malfunctions through free-text override reasons","volume":"26","author":"S Aaron","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"01","key":"ref29","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1055\/s-0039-3402757","article-title":"Reducing interruptive alert burden using quality improvement methodology","volume":"11","author":"J D Chaparro","year":"2020","journal-title":"Appl Clin Inform"},{"issue":"05","key":"ref30","doi-asserted-by":"crossref","first-page":"810","DOI":"10.1055\/s-0039-1697905","article-title":"Towards a maturity model for clinical decision support operations","volume":"10","author":"E W Orenstein","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"01","key":"ref31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1055\/s-0038-1676587","article-title":"CDS in a learning health care system: Identifying physicians' reasons for rejection of best-practice recommendations in pneumonia through computerized clinical decision support","volume":"10","author":"B E Jones","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"06","key":"ref32","doi-asserted-by":"crossref","first-page":"507","DOI":"10.3414\/ME16-01-0003","article-title":"Development of a standardized rating tool for drug alerts to reduce information overload","volume":"55","author":"B Pfistermeister","year":"2016","journal-title":"Methods Inf Med"},{"issue":"11","key":"ref33","doi-asserted-by":"crossref","first-page":"e23351","DOI":"10.2196\/23351","article-title":"Alert override patterns with a medication clinical decision support system in an academic emergency department: retrospective descriptive study","volume":"8","author":"J Yoo","year":"2020","journal-title":"JMIR Med Inform"},{"key":"ref34","doi-asserted-by":"crossref","DOI":"10.4324\/9781498757461","volume-title":"Improving Outcomes with Clinical Decision Support","author":"J Osheroff","year":"2012"},{"issue":"11","key":"ref35","doi-asserted-by":"crossref","first-page":"1552","DOI":"10.1093\/jamia\/ocy091","article-title":"Smashing the strict hierarchy: three cases of clinical decision support malfunctions involving carvedilol","volume":"25","author":"A Wright","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"06","key":"ref36","doi-asserted-by":"crossref","first-page":"709","DOI":"10.1093\/jamia\/ocy003","article-title":"Communication failure: analysis of prescribers' use of an internal free-text field on electronic prescriptions","volume":"25","author":"A Ai","year":"2018","journal-title":"J Am Med Inform Assoc"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0042-1757923.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T14:00:43Z","timestamp":1730815243000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0042-1757923"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10]]},"references-count":36,"journal-issue":{"issue":"05","published-online":{"date-parts":[[2022,10,5]]},"published-print":{"date-parts":[[2022,10]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0042-1757923","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10]]}}}