{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T01:20:49Z","timestamp":1783041649067,"version":"3.54.6"},"reference-count":65,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T00:00:00Z","timestamp":1685664000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Artificial intelligence (AI) technologies are increasingly applied to empower clinical decision support systems (CDSS), providing patient-specific recommendations to improve clinical work. Equally important to technical advancement is human, social, and contextual factors that impact the successful implementation and user adoption of AI-empowered CDSS (AI-CDSS). With the growing interest in human-centered design and evaluation of such tools, it is critical to synthesize the knowledge and experiences reported in prior work and shed light on future work.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we conducted a systematic review to gain an in-depth understanding of how AI-empowered CDSS was used, designed, and evaluated, and how clinician users perceived such systems. We performed literature search in five databases for articles published between the years 2011 and 2022. A total of 19874 articles were retrieved and screened, with 20 articles included for in-depth analysis.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The reviewed studies assessed different aspects of AI-CDSS, including effectiveness (e.g., improved patient evaluation and work efficiency), user needs (e.g., informational and technological needs), user experience (e.g., satisfaction, trust, usability, workload, and understandability), and other dimensions (e.g., the impact of AI-CDSS on workflow and patient-provider relationship). Despite the promising nature of AI-CDSS, our findings highlighted six major challenges of implementing such systems, including technical limitation, workflow misalignment, attitudinal barriers, informational barriers, usability issues, and environmental barriers. These sociotechnical challenges prevent the effective use of AI-based CDSS interventions in clinical settings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Our study highlights the paucity of studies examining the user needs, perceptions, and experiences of AI-CDSS. Based on the findings, we discuss design implications and future research directions.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fcomp.2023.1187299","type":"journal-article","created":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T12:51:42Z","timestamp":1685710302000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":90,"title":["Human-centered design and evaluation of AI-empowered clinical decision support systems: a systematic review"],"prefix":"10.3389","volume":"5","author":[{"given":"Liuping","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dakuo","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weidan","family":"Cao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaomu","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ping","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianxing","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiangmin","family":"Fan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Feng","family":"Tian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,2]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"e27992","DOI":"10.2196\/27992","article-title":"Clinical utility and functionality of an artificial intelligence\u2013based app to predict mortality in COVID-19: mixed methods analysis","volume":"5","author":"Abdulaal","year":"2021","journal-title":"JMIR Format. Res."},{"key":"B2","first-page":"1","article-title":"\u201cGuidelines for human-AI interaction,\u201d","author":"Amershi","year":"2019","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"},{"key":"B3","doi-asserted-by":"publisher","first-page":"5088","DOI":"10.3390\/app11115088","article-title":"Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: a systematic review","volume":"11","author":"Antoniadi","year":"2021","journal-title":"Appl. Sci."},{"key":"B4","article-title":"\u201cThe impact of standardized order sets on quality and financial outcomes,\u201d","author":"Ballard","year":"2008","journal-title":"Advances in Patient Safety: New Directions and Alternative Approaches (vol. 2: culture and redesign)"},{"key":"B5","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1080\/10447310802205776","article-title":"An empirical evaluation of the system usability scale","volume":"24","author":"Bangor","year":"2008","journal-title":"Intl. J. Hum. Comput. Interact."},{"key":"B6","first-page":"1","article-title":"\u201cA human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy,\u201d","author":"Beede","year":"2020","journal-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems"},{"key":"B7","doi-asserted-by":"publisher","first-page":"e22","DOI":"10.1192\/bjo.2020.127","article-title":"Using a simulation centre to evaluate preliminary acceptability and impact of an artificial intelligence-powered clinical decision support system for depression treatment on the physician\u2013patient interaction","volume":"7","author":"Benrimoh","year":"2021","journal-title":"BJPsych Open"},{"key":"B8","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1007\/s43681-022-00135-x","article-title":"Diffused responsibility: attributions of responsibility in the use of AI-driven clinical decision support systems","volume":"2","author":"Bleher","year":"2022","journal-title":"AI Ethics"},{"key":"B9","doi-asserted-by":"publisher","first-page":"29","DOI":"10.7326\/0003-4819-157-1-201207030-00450","article-title":"Effect of clinical decision-support systems: a systematic review","volume":"157","author":"Bright","year":"2012","journal-title":"Ann. Intern. Med."},{"key":"B10","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1016\/j.ijmedinf.2017.02.014","article-title":"A web-based clinical decision support system for gestational diabetes: Automatic diet prescription and detection of insulin needs","volume":"102","author":"Caballero-Ruiz","year":"2017","journal-title":"Int. J. Med. Inform."},{"key":"B11","first-page":"1","article-title":"\u201cHuman-centered tools for coping with imperfect algorithms during medical decision-making,\u201d","author":"Cai","year":"","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359206","article-title":"\u201cHello AI\u201d: uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making","volume":"3","author":"Cai","year":"","journal-title":"Proc. ACM Hum. Comput. Interact."},{"key":"B13","doi-asserted-by":"publisher","first-page":"1757","DOI":"10.1093\/jamia\/ocac110","article-title":"Design, effectiveness, and economic outcomes of contemporary chronic disease clinical decision support systems: a systematic review and meta-analysis","volume":"29","author":"Chen","year":"2022","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B14","first-page":"1","article-title":"\u201cExplaining decision-making algorithms through UI: Strategies to help non-expert stakeholders,\u201d","author":"Cheng","year":"2019","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"},{"key":"B15","first-page":"89","article-title":"Physician usage and acceptance of a machine learning recommender system for simulated clinical order entry","volume":"2020","author":"Chiang","year":"2020","journal-title":"AMIA Summits Translat. Sci. Proc."},{"key":"B16","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1148\/rg.2017160130","article-title":"Machine learning for medical imaging","volume":"37","author":"Erickson","year":"2017","journal-title":"Radiographics"},{"key":"B17","doi-asserted-by":"publisher","first-page":"e19928","DOI":"10.2196\/19928","article-title":"Utilization of self-diagnosis health chatbots in real-world settings: case study","volume":"23","author":"Fan","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"B18","volume-title":"Ethnography: Step-by-Step.","author":"Fetterman","year":"2019"},{"key":"B19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6947-13-135","article-title":"Systematic review of clinical decision support interventions with potential for inpatient cost reduction","volume":"13","author":"Fillmore","year":"2013","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"B20","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1609\/aimag.v38i3.2741","article-title":"European Union regulations on algorithmic decision-making and a \u201cright to explanation\u201d","volume":"38","author":"Goodman","year":"2017","journal-title":"AI Magazine"},{"key":"B21","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1097\/00115514-201101000-00006","article-title":"Safety issues related to the electronic medical record (EMR): synthesis of the literature from the last decade, 2000-2009","volume":"56","author":"Harrington","year":"2011","journal-title":"J. Healthcare Manage."},{"key":"B22","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): results of empirical and theoretical research","volume":"52","author":"Hart","year":"1988","journal-title":"Adv. Psychol."},{"key":"B23","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/13373.001.0001","volume-title":"How Humans Judge Machines.","author":"Hidalgo","year":"2021"},{"key":"B24","first-page":"1","article-title":"\u201cThe clinical decision support system for the snake envenomation in Thailand,\u201d","author":"Hoonlor","year":"2018","journal-title":"2018 15th International Joint Conference on Computer Science and Software Engineering (JCSSE)."},{"key":"B25","first-page":"1","article-title":"\u201cDesigning AI for trust and collaboration in time-constrained medical decisions: a sociotechnical lens,\u201d","author":"Jacobs","year":"2021","journal-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10916-021-01727-6","article-title":"Technology acceptance of a machine learning algorithm predicting delirium in a clinical setting: a mixed-methods study","volume":"45","author":"Jauk","year":"2021","journal-title":"J. Med. Syst."},{"key":"B27","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1055\/s-0041-1736461","article-title":"Status of AI-enabled clinical decision support systems implementations in China","volume":"60","author":"Ji","year":"2021","journal-title":"Methods Inf. Med."},{"key":"B28","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1136\/svn-2017-000101","article-title":"Artificial intelligence in healthcare: past, present and future","volume":"2","author":"Jiang","year":"2017","journal-title":"Stroke Vasc. Neurol."},{"key":"B29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3344258","article-title":"Carepre: an intelligent clinical decision assistance system","volume":"1","author":"Jin","year":"2020","journal-title":"ACM Trans. Comput. Healthcare"},{"key":"B30","doi-asserted-by":"publisher","first-page":"e210013","DOI":"10.1148\/ryai.2021210013","article-title":"Integrating Al algorithms into the clinical workflow","volume":"3","author":"Juluru","year":"2021","journal-title":"Radiol. Artif. Intelligence"},{"key":"B31","doi-asserted-by":"publisher","first-page":"e28639","DOI":"10.2196\/28639","article-title":"Human factors and technological characteristics influencing the interaction of medical professionals with artificial intelligence\u2013enabled clinical decision support systems: literature review","volume":"9","author":"Knop","year":"2022","journal-title":"JMIR Human Factors"},{"key":"B32","doi-asserted-by":"publisher","first-page":"1850","DOI":"10.1093\/jamia\/ocaa190","article-title":"OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases","volume":"27","author":"Kumar","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B33","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1109\/TE.2013.2292570","article-title":"Performing systematic literature reviews with novices: an iterative approach","volume":"57","author":"Lavall\u00e9e","year":"2013","journal-title":"IEEE Trans. Educ."},{"key":"B34","first-page":"1","article-title":"\u201cA human-ai collaborative approach for clinical decision making on rehabilitation assessment,\u201d","author":"Lee","year":"2021","journal-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12913-020-05688-3","article-title":"Economic impact of clinical decision support interventions based on electronic health records","volume":"20","author":"Lewkowicz","year":"2020","journal-title":"BMC Health Serv. Res."},{"key":"B36","doi-asserted-by":"publisher","first-page":"1700","DOI":"10.1177\/1460458219889794","article-title":"The efficiency analysis and spatial implications of health information technology: a regional exploratory study in China","volume":"26","author":"Li","year":"2020","journal-title":"Health Informatics J."},{"key":"B37","doi-asserted-by":"publisher","first-page":"11591","DOI":"10.1073\/pnas.1806905115","article-title":"Deep neural network improves fracture detection by clinicians","volume":"115","author":"Lindsey","year":"2018","journal-title":"Proc. Nat. Acad. Sci."},{"key":"B38","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1055\/s-0039-1677903","article-title":"Artificial intelligence in clinical decision support: challenges for evaluating AI and practical implications","volume":"28","author":"Magrabi","year":"2019","journal-title":"Yearb. Med. Inform."},{"key":"B39","doi-asserted-by":"publisher","first-page":"709","DOI":"10.2307\/258792","article-title":"An integrative model of organizational trust","volume":"20","author":"Mayer","year":"1995","journal-title":"Acad. Manage. Rev."},{"key":"B40","doi-asserted-by":"publisher","first-page":"264","DOI":"10.7326\/0003-4819-151-4-200908180-00135","article-title":"Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement","volume":"151","author":"Moher","year":"2009","journal-title":"Ann. Intern. Med."},{"key":"B41","doi-asserted-by":"publisher","first-page":"1326","DOI":"10.1080\/20479700.2020.1758894","article-title":"Addressing power dynamics in interprofessional health care teams","volume":"14","author":"Okpala","year":"2021","journal-title":"Int. J. Healthcare Manage."},{"key":"B42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10916-014-0099-9","article-title":"Understanding critical barriers to implementing a clinical information system in a nursing home through the lens of a socio-technical perspective","volume":"38","author":"Or","year":"2014","journal-title":"J. Med. Syst."},{"key":"B43","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/j.artmed.2008.07.017","article-title":"The coming of age of artificial intelligence in medicine","volume":"46","author":"Patel","year":"2009","journal-title":"Artif. Intell. Med."},{"key":"B44","doi-asserted-by":"publisher","first-page":"104072","DOI":"10.1016\/j.ijmedinf.2019.104072","article-title":"A lesson in implementation: a pre-post study of providers' experience with artificial intelligence-based clinical decision support","volume":"137","author":"Romero-Brufau","year":"2020","journal-title":"Int. J. Med. Inform."},{"key":"B45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6947-12-142","article-title":"Intention to adopt clinical decision support systems in a developing country: effect of Physician's perceived professional autonomy, involvement and belief: a cross-sectional study","volume":"12","author":"Sambasivan","year":"2012","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"B46","doi-asserted-by":"crossref","first-page":"102684","DOI":"10.1016\/j.ijhcs.2021.102684","article-title":"Human-centered XAI: Developing design patterns for explanations of clinical decision support systems","volume":"154","author":"Schoonderwoerd","year":"2021","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"B47","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1145\/3351095.3372827","article-title":"\u201c\u201cThe human body is a black box\u201d supporting clinical decision-making with deep learning,\u201d","author":"Sendak","year":"2020","journal-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency"},{"key":"B48","doi-asserted-by":"publisher","first-page":"2199","DOI":"10.1001\/jama.2018.17163","article-title":"Clinical decision support in the era of artificial intelligence","volume":"320","author":"Shortliffe","year":"2018","journal-title":"JAMA"},{"key":"B49","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/978-3-319-17272-9_4","article-title":"\u201cA new socio-technical model for studying health information technology in complex adaptive healthcare systems,\u201d","author":"Sittig","year":"2015","journal-title":"Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare"},{"key":"B50","volume-title":"The Ethnographic Interview.","author":"Spradley","year":""},{"key":"B51","volume-title":"Participant Observation.","author":"Spradley","year":""},{"key":"B52","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1145\/2110363.2110423","article-title":"\u201cSmart alarms: multivariate medical alarm integration for post CABG surgery patients,\u201d","author":"Stevens","year":"2012","journal-title":"Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium"},{"key":"B53","doi-asserted-by":"publisher","first-page":"e256","DOI":"10.1136\/bmj.e256","article-title":"Shared decision making: really putting patients at the centre of healthcare","volume":"344","author":"Stiggelbout","year":"2012","journal-title":"BMJ"},{"key":"B54","article-title":"Artificial intelligence and life in 2030: the one hundred year study on artificial intelligence","author":"Stone","year":"2022","journal-title":"arXiv preprint."},{"key":"B55","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1109\/MSPEC.2019.8678513","article-title":"IBM Watson, heal thyself: How IBM overpromised and underdelivered on AI health care","volume":"56","author":"Strickland","year":"2019","journal-title":"IEEE Spectrum"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.1145\/3544549.3583178","article-title":"Toward human-centered responsible artificial intelligence: a review of CHI research and industry toolkits","author":"Tahaei","year":"2023","journal-title":"arXiv preprint."},{"key":"B57","doi-asserted-by":"publisher","first-page":"114336","DOI":"10.1016\/j.psychres.2021.114336","article-title":"Evaluating the perceived utility of an artificial intelligence-powered clinical decision support system for depression treatment using a simulation center","volume":"308","author":"Tanguay-Sela","year":"2022","journal-title":"Psychiatry Res."},{"key":"B58","first-page":"867","article-title":"Health information technology in primary health care in developing countries: a literature review","volume":"82","author":"Tomasi","year":"2004","journal-title":"Bull. World Health Organ."},{"key":"B59","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1197\/jamia.M2378","article-title":"Technology implementation and workarounds in the nursing home","volume":"15","author":"Vogelsmeier","year":"2008","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B60","first-page":"1","article-title":"\u201c\u201cBrilliant AI doctor\u201d in rural clinics: challenges in AI-powered clinical decision support system deployment,\u201d","author":"Wang","year":"2021","journal-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"},{"key":"B61","first-page":"1","article-title":"\u201cUnremarkable AI: fitting intelligent decision support into critical, clinical decision-making processes,\u201d","author":"Yang","year":"2019","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"},{"key":"B62","doi-asserted-by":"crossref","first-page":"4477","DOI":"10.1145\/2858036.2858373","article-title":"\u201cInvestigating the heart pump implant decision process: opportunities for decision support tools to help,\u201d","author":"Yang","year":"2016","journal-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems"},{"key":"B63","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1007\/s10916-021-01743-6","article-title":"Effect of ai explanations on human perceptions of patient-facing ai-powered healthcare systems","volume":"45","author":"Zhang","year":"2021","journal-title":"J. Med. Syst."},{"key":"B64","doi-asserted-by":"publisher","first-page":"ooac113","DOI":"10.1093\/jamiaopen\/ooac113","article-title":"Designing and implementing smart glass technology for emergency medical services: a sociotechnical perspective","volume":"5","author":"Zhang","year":"2022","journal-title":"JAMIA Open"},{"key":"B65","doi-asserted-by":"publisher","first-page":"e0231166","DOI":"10.1371\/journal.pone.0231166","article-title":"Opening the black box of artificial intelligence for clinical decision support: a study predicting stroke outcome","volume":"15","author":"Zihni","year":"2020","journal-title":"PLoS ONE"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2023.1187299\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T12:52:07Z","timestamp":1685710327000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2023.1187299\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,2]]},"references-count":65,"alternative-id":["10.3389\/fcomp.2023.1187299"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2023.1187299","relation":{},"ISSN":["2624-9898"],"issn-type":[{"value":"2624-9898","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,2]]},"article-number":"1187299"}}