{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T09:24:12Z","timestamp":1758273852209,"version":"3.37.3"},"reference-count":40,"publisher":"Georg Thieme Verlag KG","issue":"01","funder":[{"DOI":"10.13039\/100000133","name":"Agency for Healthcare Research and Quality","doi-asserted-by":"crossref","award":["R18HS027735"],"award-info":[{"award-number":["R18HS027735"]}],"id":[{"id":"10.13039\/100000133","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Background\u2003Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.<\/jats:p><jats:p>\n          Objectives\u2003In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.<\/jats:p><jats:p>\n          Methods\u2003We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.<\/jats:p><jats:p>\n          Results\u2003Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.<\/jats:p><jats:p>\n          Conclusion\u2003We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.<\/jats:p>","DOI":"10.1055\/a-2219-5175","type":"journal-article","created":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T00:11:15Z","timestamp":1701303075000},"page":"164-169","source":"Crossref","is-referenced-by-count":2,"title":["Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions"],"prefix":"10.1055","volume":"15","clinical-trial-number":[{"clinical-trial-number":"nct05810064","registry":"10.18810\/clinical-trials-gov"}],"author":[{"given":"Daniel J.","family":"Hekman","sequence":"additional","affiliation":[{"name":"Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hanna J.","family":"Barton","sequence":"additional","affiliation":[{"name":"Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Apoorva P.","family":"Maru","sequence":"additional","affiliation":[{"name":"Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Graham","family":"Wills","sequence":"additional","affiliation":[{"name":"Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amy L.","family":"Cochran","sequence":"additional","affiliation":[{"name":"Department of Population Health, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Corey","family":"Fritsch","sequence":"additional","affiliation":[{"name":"Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Douglas A.","family":"Wiegmann","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Frank","family":"Liao","sequence":"additional","affiliation":[{"name":"Department of Applied Data Science, UWHealth Hospitals and Clinics, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brian W.","family":"Patterson","sequence":"additional","affiliation":[{"name":"Berbee-Walsh Department of Emergency Medicine, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States"},{"name":"Department of Population Health, University of Wisconsin-Madison, School of Medicine and Public Health, Madison, Wisconsin, United States"},{"name":"Department of Industrial and Systems Engineering, University of Wisconsin-Madison, Madison, Wisconsin, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"194","published-online":{"date-parts":[[2024,2,28]]},"reference":[{"issue":"07","key":"ref1","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1377\/hlthaff.2014.0041","article-title":"Big data in health care: using analytics to identify and manage high-risk and high-cost patients","volume":"33","author":"D W Bates","year":"2014","journal-title":"Health Aff (Millwood)"},{"issue":"26","key":"ref2","doi-asserted-by":"crossref","first-page":"2507","DOI":"10.1056\/NEJMp1702071","article-title":"Machine learning and prediction in medicine: beyond the peak of inflated expectations","volume":"376","author":"J H Chen","year":"2017","journal-title":"N Engl J Med"},{"issue":"13","key":"ref3","doi-asserted-by":"crossref","first-page":"1216","DOI":"10.1056\/NEJMp1606181","article-title":"Predicting the future: big data, machine learning, and clinical medicine","volume":"375","author":"Z Obermeyer","year":"2016","journal-title":"N Engl J Med"},{"issue":"01","key":"ref4","doi-asserted-by":"crossref","first-page":"e14512","DOI":"10.2196\/14512","article-title":"Back to the future: achieving health equity through health informatics and digital health","volume":"8","author":"L C Brewer","year":"2020","journal-title":"JMIR Mhealth Uhealth"},{"issue":"08","key":"ref5","doi-asserted-by":"crossref","first-page":"1515","DOI":"10.1016\/j.ajem.2018.01.017","article-title":"How artificial intelligence could transform emergency department operations","volume":"36","author":"Y Berlyand","year":"2018","journal-title":"Am J Emerg Med"},{"key":"ref6","doi-asserted-by":"crossref","first-page":"103498","DOI":"10.1016\/j.apergo.2021.103498","article-title":"Workflow integration analysis of a human factors-based clinical decision support in the emergency department","volume":"97","author":"M E Salwei","year":"2021","journal-title":"Appl Ergon"},{"issue":"03","key":"ref7","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1377\/hlthaff.27.3.759","article-title":"The triple aim: care, health, and cost","volume":"27","author":"D M Berwick","year":"2008","journal-title":"Health Aff (Millwood)"},{"issue":"10","key":"ref8","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1136\/bmjqs-2015-004160","article-title":"The quadruple aim: care, health, cost and meaning in work","volume":"24","author":"R Sikka","year":"2015","journal-title":"BMJ Qual Saf"},{"issue":"06","key":"ref9","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1001\/jama.2021.25181","article-title":"The quintuple aim for health care improvement: a new imperative to advance health equity","volume":"327","author":"S Nundy","year":"2022","journal-title":"JAMA"},{"key":"ref10","doi-asserted-by":"crossref","first-page":"103033","DOI":"10.1016\/j.apergo.2019.103033","article-title":"SEIPS 3.0: Human-centered design of the patient journey for patient safety","volume":"84","author":"P Carayon","year":"2020","journal-title":"Appl Ergon"},{"issue":"01","key":"ref11","doi-asserted-by":"crossref","first-page":"100598","DOI":"10.1016\/j.hjdsi.2021.100598","article-title":"Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments","volume":"10","author":"G C Jacobsohn","year":"2022","journal-title":"Healthc (Amst)"},{"issue":"01","key":"ref12","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1186\/s12911-017-0430-8","article-title":"Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system","volume":"17","author":"J S Ancker","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"issue":"01","key":"ref13","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1055\/s-0040-1701257","article-title":"Heart failure dashboard design and validation to improve care of veterans","volume":"11","author":"M Foster","year":"2020","journal-title":"Appl Clin Inform"},{"issue":"02","key":"ref14","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1055\/s-0042-1744387","article-title":"Visualizing opioid-use variation in a pediatric perioperative dashboard","volume":"13","author":"C W Safranek","year":"2022","journal-title":"Appl Clin Inform"},{"issue":"04","key":"ref15","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1055\/s-0039-1693712","article-title":"A visual analytics dashboard to summarize serial anesthesia records in pediatric radiation treatment","volume":"10","author":"O Nelson","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"03","key":"ref16","doi-asserted-by":"crossref","first-page":"711","DOI":"10.4338\/ACI-2016-03-RA-0049","article-title":"Visual analytics for pattern discovery in home care","volume":"7","author":"K Radhakrishnan","year":"2016","journal-title":"Appl Clin Inform"},{"issue":"02","key":"ref17","doi-asserted-by":"crossref","first-page":"E21","DOI":"10.1097\/HMR.0000000000000324","article-title":"Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation","volume":"47","author":"S J Singer","year":"2022","journal-title":"Health Care Manage Rev"},{"issue":"01","key":"ref18","doi-asserted-by":"crossref","first-page":"23017","DOI":"10.1038\/s41598-021-02481-y","article-title":"Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19","volume":"11","author":"C Duckworth","year":"2021","journal-title":"Sci Rep"},{"issue":"01","key":"ref19","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1038\/s41746-021-00468-7","article-title":"A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission","volume":"4","author":"V V Mi\u0161i\u0107","year":"2021","journal-title":"NPJ Digit Med"},{"key":"ref20","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1007\/s10115-008-0139-1","article-title":"A framework for monitoring classifiers' performance: When and why failure occurs?","volume":"18","author":"D Cieslak","year":"2009","journal-title":"Knowl Inf Syst"},{"issue":"09","key":"ref21","doi-asserted-by":"crossref","first-page":"1631","DOI":"10.1093\/jamia\/ocac078","article-title":"A framework for the oversight and local deployment of safe and high-quality prediction models","volume":"29","author":"A D Bedoya","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"01","key":"ref22","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1038\/s41746-022-00611-y","article-title":"Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare","volume":"5","author":"J Feng","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"07","key":"ref23","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1097\/MLR.0000000000001140","article-title":"Training and interpreting machine learning algorithms to evaluate fall risk after emergency department visits","volume":"57","author":"B W Patterson","year":"2019","journal-title":"Med Care"},{"issue":"03","key":"ref25","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1055\/s-0038-1666842","article-title":"The development of heuristics for evaluation of dashboard visualizations","volume":"9","author":"D Dowding","year":"2018","journal-title":"Appl Clin Inform"},{"issue":"05","key":"ref26","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.clml.2019.10.002","article-title":"Treatment journeys of patients with newly diagnosed multiple myeloma (NDMM): results from the Connect MM Registry","volume":"20","author":"S Jagannath","year":"2020","journal-title":"Clin Lymphoma Myeloma Leuk"},{"issue":"01","key":"ref27","doi-asserted-by":"crossref","first-page":"31","DOI":"10.3102\/10769986027001031","article-title":"Visions and re-visions of Charles Joseph Minard","volume":"27","author":"M Friendly","year":"2002","journal-title":"J Educ Behav Stat"},{"key":"ref28","doi-asserted-by":"crossref","first-page":"221","DOI":"10.2147\/JHL.S319169","article-title":"The contribution of variable control charts to quality improvement in healthcare: a literature review","volume":"13","author":"L Slyngstad","year":"2021","journal-title":"J Healthc Leadersh"},{"issue":"05","key":"ref29","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1016\/j.jemermed.2014.12.019","article-title":"Statistical process control: separating signal from noise in emergency department operations","volume":"48","author":"L Pimentel","year":"2015","journal-title":"J Emerg Med"},{"issue":"12","key":"ref30","doi-asserted-by":"crossref","first-page":"1212","DOI":"10.1097\/00005650-199112000-00004","article-title":"Controlling variation in health care: a consultation from Walter Shewhart","volume":"29","author":"D M Berwick","year":"1991","journal-title":"Med Care"},{"issue":"05","key":"ref31","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1136\/qshc.2006.022194","article-title":"Application of statistical process control in healthcare improvement: systematic review","volume":"16","author":"J Thor","year":"2007","journal-title":"Qual Saf Health Care"},{"issue":"03","key":"ref32","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1109\/52.922728","article-title":"Heuristics for iterative software development","volume":"18","author":"D Sotirovski","year":"2001","journal-title":"IEEE Softw"},{"key":"ref33","first-page":"528","article-title":"The goal question metric approach","author":"V RBG Caldiera","year":"1994","journal-title":"Encycl Softw Eng"},{"issue":"43","key":"ref38","doi-asserted-by":"crossref","first-page":"1686","DOI":"10.21105\/joss.01686","article-title":"Welcome to the tidyverse","volume":"4","author":"H Wickham","year":"2019","journal-title":"J Open Source Softw"},{"volume-title":"The Grammar of Graphics. 2nd ed","year":"2005","author":"L Wilkinson","key":"ref39"},{"issue":"02","key":"ref40","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.annemergmed.2011.04.008","article-title":"Understanding better how clinicians work","volume":"58","author":"R J Fairbanks","year":"2011","journal-title":"Ann Emerg Med"},{"issue":"12","key":"ref41","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.1001\/jama.1988.03410120089033","article-title":"The quality of care. How can it be assessed?","volume":"260","author":"A Donabedian","year":"1988","journal-title":"JAMA"},{"issue":"S 01","key":"ref42","doi-asserted-by":"crossref","first-page":"i50","DOI":"10.1136\/qshc.2005.015842","article-title":"Work system design for patient safety: the SEIPS model","volume":"15","author":"P Carayon","year":"2006","journal-title":"Qual Saf Health Care"},{"issue":"03","key":"ref43","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1510\/icvts.2009.204768","article-title":"Control charts, Cusum techniques and funnel plots. A review of methods for monitoring performance in healthcare","volume":"9","author":"L Noyez","year":"2009","journal-title":"Interact Cardiovasc Thorac Surg"},{"key":"ref44","doi-asserted-by":"crossref","first-page":"e48128","DOI":"10.2196\/48128","article-title":"Effectiveness of an emergency department-based machine learning clinical decision support tool to prevent outpatient falls among older adults: protocol for a quasi-experimental study","volume":"12","author":"D J Hekman","year":"2023","journal-title":"JMIR Res Protoc"},{"volume-title":"Quality-control handbook","year":"1951","author":"J M Juran","key":"ref45"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/a-2219-5175.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T14:04:30Z","timestamp":1730815470000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/a-2219-5175"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,29]]},"references-count":40,"journal-issue":{"issue":"01","published-online":{"date-parts":[[2024,1,3]]},"published-print":{"date-parts":[[2024,1]]}},"URL":"https:\/\/doi.org\/10.1055\/a-2219-5175","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"type":"electronic","value":"1869-0327"}],"subject":[],"published-other":{"date-parts":[[2023,11,29]]},"published":{"date-parts":[[2023,11,29]]}}}