{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T05:55:44Z","timestamp":1790920544855,"version":"4.1.0"},"reference-count":57,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Healthcare artificial intelligence has produced substantial diagnostic capabilities, yet clinical operations remain underserved by AI transparency research. Existing reviews of Explainable AI (XAI) in healthcare concentrate on diagnostic imaging, oncology, radiology, and clinical decision support. The governance needs of clinical process intelligence (CPI) systems, which support patient flow, resource allocation, throughput management, and care coordination, remain comparatively underdeveloped.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This original research article reports a structured literature review with systematic search and screening, followed by framework development. It characterizes healthcare XAI research across application domains and introduces the CPI-XAI Governance Framework, a theory-informed model for operationalizing algorithmic transparency as an organizational information-systems capability in clinical process intelligence environments.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We searched PubMed\/MEDLINE, Scopus, Web of Science, IEEE Xplore, ACM Digital Library, and CINAHL for peer-reviewed studies published from January 2015 to April 2026. Screening and extraction used transparent procedures adapted from PRISMA flow-reporting practices. Two authors independently screened records and extracted domain, XAI method, setting, evaluation, and governance variables; disagreements were resolved by author adjudication. Framework development used a documented hybrid deductive-inductive coding pipeline, source-contribution rules, negative-case review, and an auditable evidence-to-framework mapping anchored in IT Capability Theory, the DeLone\u2013McLean IS Success Model, implementation science, sociotechnical systems theory, and health AI governance guidance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Of 187 included studies, 175 (93.6%) addressed diagnostic, disease-specific, imaging, or general clinical decision-support contexts, while operational and workflow-embedded AI accounted for 9 studies (4.8%). The primary result is descriptive: XAI research in healthcare remains concentrated in diagnostic domains, while explicit explainability-governance work for clinical operations is sparse within the search boundary. Five evidence gaps informed the framework: limited operational XAI evaluation standards, role-blind explanation design, weak causal and counterfactual support, poor workflow embedding, and absence of longitudinal XAI governance. A preliminary 0\u201315 scoring rubric and two hypothetical organizational assessments demonstrate how the framework can be applied analytically; they do not constitute external validation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Healthcare XAI remains concentrated in diagnostic and clinical decision-support contexts, while explicit explainability-governance research for operational AI systems remains thinly represented. The CPI-XAI Governance Framework addresses this gap by treating explainability as an organizational capability rather than a model-level technical feature. The framework is preliminary and theory-informed; it should be empirically validated before use as a measurement instrument.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2026.1864565","type":"journal-article","created":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T05:36:17Z","timestamp":1790919377000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Beyond the black box in healthcare operations: a structured literature review and governance framework for explainable AI in clinical process intelligence systems"],"prefix":"10.3389","volume":"9","author":[{"given":"Albert Adusei","family":"Brobbey","sequence":"first","affiliation":[{"name":"California InterContinental University","place":["Sioux Falls, SD, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Narayan","family":"Bhosale","sequence":"additional","affiliation":[{"name":"California InterContinental University","place":["Sioux Falls, SD, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Ackom","sequence":"additional","affiliation":[{"name":"California InterContinental University","place":["Sioux Falls, SD, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,10,2]]},"reference":[{"key":"ref1","first-page":"1","author":"Abdul","year":"2018"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/access.2018.2870052","article-title":"Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"20552076251368045","DOI":"10.1177\/20552076251368045","article-title":"XAI-Eval: a framework for comparative evaluation of explanation methods in healthcare","volume":"11","author":"Agrawal","year":"2025","journal-title":"Digit. Health"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"1749527","DOI":"10.3389\/frai.2026.1749527","article-title":"Explainable AI in healthcare: a systematic review of XAI use cases in imaging, diagnostics, and rehabilitation","volume":"9","author":"Aravindkumar","year":"2026","journal-title":"Front. Artif. Intell."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1080\/1364557032000119616","article-title":"Scoping studies: towards a methodological framework","volume":"8","author":"Arksey","year":"2005","journal-title":"Int. J. Soc. Res. Methodol."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Arrieta","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"e63733","DOI":"10.2196\/63733","article-title":"Designing clinical decision support systems (CDSS): a user-centered lens of the design characteristics, challenges, and implications","volume":"27","author":"Bayor","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"169","DOI":"10.2307\/3250983","article-title":"A resource-based perspective on information technology capability and firm performance: an empirical investigation","volume":"24","author":"Bharadwaj","year":"2000","journal-title":"MIS Q."},{"key":"ref9","author":"Chen","year":"2020"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1186\/1748-5908-4-50","article-title":"Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science","volume":"4","author":"Damschroder","year":"2009","journal-title":"Implement. Sci."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1080\/07421222.2003.11045748","article-title":"The DeLone and McLean model of information systems success: a ten-year update","volume":"19","author":"DeLone","year":"2003","journal-title":"J. Manag. Inf. Syst."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1186\/s12910-024-01103-2","article-title":"The ethical requirement of explainability for AI-DSS in healthcare: a systematic review of reasons","volume":"25","author":"Freyer","year":"2024","journal-title":"BMC Med. Ethics"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","year":"2021","journal-title":"Lancet Digit. Health"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"e367","DOI":"10.2196\/jmir.8775","article-title":"Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies","volume":"19","author":"Greenhalgh","year":"2017","journal-title":"J. Med. Internet Res."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"611","DOI":"10.2307\/25148742","article-title":"The nature of theory in information systems","volume":"30","author":"Gregor","year":"2006","journal-title":"MIS Q."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"e1230","DOI":"10.1002\/cl2.1288","article-title":"PRISMA2020: an R package and shiny app for producing PRISMA 2020-compliant flow diagrams","volume":"18","author":"Haddaway","year":"2022","journal-title":"Campbell Syst. Rev."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1186\/s13012-016-0398-2","article-title":"PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice","volume":"11","author":"Harvey","year":"2016","journal-title":"Implement. Sci."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"1399","DOI":"10.3390\/diagnostics15111399","article-title":"Explainable artificial intelligence in radiological cardiovascular imaging: a systematic review","volume":"15","author":"Haupt","year":"2025","journal-title":"Diagnostics (Basel)"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"e1312","DOI":"10.1002\/widm.1312","article-title":"Causability and explainability of artificial intelligence in medicine","volume":"9","author":"Holzinger","year":"2019","journal-title":"Wiley Interdiscip Rev Data Min Knowl Discov."},{"key":"ref20","doi-asserted-by":"publisher","first-page":"285","DOI":"10.3233\/EFI-180221","article-title":"The mixed methods appraisal tool (MMAT) version 2018 for information professionals and researchers","volume":"34","author":"Hong","year":"2018","journal-title":"Educ. Inf."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1186\/s12916-019-1426-2","article-title":"Key challenges for delivering clinical impact with artificial intelligence","volume":"17","author":"Kelly","year":"2019","journal-title":"BMC Med."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/1748-5908-5-69","article-title":"Scoping studies: advancing the methodology","volume":"5","author":"Levac","year":"2010","journal-title":"Implement. Sci."},{"key":"ref24","author":"Liao","year":"2021"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"e73374","DOI":"10.2196\/73374","article-title":"Improving explainability and integrability of medical AI to promote health care professional acceptance and use: mixed systematic review","volume":"27","author":"Lin","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref26","first-page":"30","author":"Lundberg","year":"2017"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"103655","DOI":"10.1016\/j.jbi.2020.103655","article-title":"The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies","volume":"113","author":"Markus","year":"2021","journal-title":"J. Biomed. Inform."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in artificial intelligence: insights from the social sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artif. Intell."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: a comprehensive review","volume":"55","author":"Minh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"542","DOI":"10.1016\/j.csbj.2024.08.005","article-title":"Unveiling the black box: a systematic review of explainable artificial intelligence in medical image analysis","volume":"24","author":"Muhammad","year":"2024","journal-title":"Comput. Struct. Biotechnol. J."},{"key":"ref31","doi-asserted-by":"publisher","first-page":"20","DOI":"10.21037\/jhmhp-24-138","article-title":"Integrating artificial intelligence in clinical practice, hospital management, and health policy: literature review","volume":"9","author":"Nasef","year":"2025","journal-title":"J. Hosp. Manag. Health Policy"},{"key":"ref32","volume-title":"Artificial Intelligence risk Management Framework (AI RMF 1.0)","year":"2023"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"e70018","DOI":"10.1002\/widm.70018","article-title":"Unveiling explainable AI in healthcare: current trends, challenges, and future directions","volume":"15","author":"Noor","year":"2025","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"key":"ref35","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1186\/s13643-021-01626-4","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"10","author":"Page","year":"2020","journal-title":"Syst. Rev."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1007\/s10729-024-09682-7","article-title":"A systematic literature review of predicting patient discharges using statistical methods","volume":"27","author":"Pahlevani","year":"2024","journal-title":"Health Care Manag. Sci."},{"key":"ref37","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41746-019-0155-4","article-title":"The 'inconvenient truth' about AI in healthcare","volume":"2","author":"Panch","year":"2019","journal-title":"NPJ Digital Medicine."},{"key":"ref38","doi-asserted-by":"publisher","first-page":"105810","DOI":"10.1016\/j.ijmedinf.2025.105810","article-title":"Human-centred AI for emergency cardiac care: evaluating RAPIDx AI with PROLIFERATE_AI","volume":"196","author":"Pinero de Plaza","year":"2025","journal-title":"Int. J. Med. Inform."},{"key":"ref39","doi-asserted-by":"publisher","first-page":"1154614","DOI":"10.3389\/frhs.2023.1154614","article-title":"Co-designing, measuring, and optimizing innovations and solutions within complex adaptive health systems","volume":"3","author":"Pinero de Plaza","year":"2023","journal-title":"Front. Health Serv."},{"key":"ref40","doi-asserted-by":"publisher","first-page":"1627169","DOI":"10.3389\/fradi.2025","article-title":"Explainable AI in medicine: challenges of integrating XAI into the future clinical routine","volume":"5","author":"Ratz","year":"2025","journal-title":"Front. Radiol."},{"key":"ref41","first-page":"1135","author":"Ribeiro","year":"2016"},{"key":"ref42","doi-asserted-by":"publisher","first-page":"e53207","DOI":"10.2196\/53207","article-title":"How explainable artificial intelligence can increase or decrease clinicians' trust in AI applications in health care: systematic review","volume":"3","author":"Rosenbacke","year":"2024","journal-title":"JMIR AI"},{"key":"ref43","doi-asserted-by":"publisher","first-page":"109370","DOI":"10.1016\/j.compeleceng.2024.109370","article-title":"A review of explainable artificial intelligence in healthcare","volume":"118","author":"Sadeghi","year":"2024","journal-title":"Comput. Electr. Eng."},{"key":"ref44","doi-asserted-by":"publisher","first-page":"119","DOI":"10.3390\/informatics12040119","article-title":"Explainable AI for clinical decision support systems: literature review, key gaps, and research synthesis","volume":"12","author":"Salimparsa","year":"2025","journal-title":"Informatics"},{"key":"ref45","first-page":"618","author":"Selvaraju","year":"2017"},{"key":"ref46","doi-asserted-by":"publisher","first-page":"e50048","DOI":"10.2196\/50048","article-title":"Toward fairness, accountability, transparency, and ethics in AI for social media and health care: scoping review","volume":"12","author":"Singhal","year":"2024","journal-title":"JMIR Med. Inform."},{"key":"ref47","doi-asserted-by":"publisher","first-page":"i68","DOI":"10.1136\/qshc.2010.042085","article-title":"A new sociotechnical model for studying health information technology in complex adaptive healthcare systems","volume":"19","author":"Sittig","year":"2010","journal-title":"Qual. Saf. Health Care"},{"key":"ref48","author":"Tonekaboni","year":"2019"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat. Med."},{"key":"ref50","doi-asserted-by":"publisher","first-page":"467","DOI":"10.7326\/M18-0850","article-title":"PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation","volume":"169","author":"Tricco","year":"2018","journal-title":"Ann. Intern. Med."},{"key":"ref51","volume-title":"Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. Draft Guidance for Industry and FDA staff","year":""},{"key":"ref52","volume-title":"Artificial Intelligence in Software as a Medical Device","year":""},{"key":"ref53","doi-asserted-by":"publisher","first-page":"102470","DOI":"10.1016\/j.media.2022.102470","article-title":"Explainable artificial intelligence (XAI) in deep learning-based medical image analysis","volume":"79","author":"van der Velden","year":"2022","journal-title":"Med. Image Anal."},{"key":"ref54","doi-asserted-by":"publisher","first-page":"841","DOI":"10.2139\/ssrn.3063289","article-title":"Counterfactual explanations without opening the black box: automated decisions and the GDPR","volume":"31","author":"Wachter","year":"2017","journal-title":"Harv. J. Law Technol."},{"key":"ref55","volume-title":"Ethics and Governance of artificial Intelligence for Health: WHO Guidance","year":"2021"},{"key":"ref56","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1186\/s13690-023-01029-y","article-title":"The relationship between hospital ownership, in-hospital mortality, and medical expenses","volume":"81","author":"Xue","year":"2023","journal-title":"Arch. Public Health"},{"key":"ref57","doi-asserted-by":"publisher","first-page":"237","DOI":"10.3390\/diagnostics12020237","article-title":"Applications of explainable artificial intelligence in diagnosis and surgery","volume":"12","author":"Zhang","year":"2022","journal-title":"Diagnostics."}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1864565\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T05:36:18Z","timestamp":1790919378000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1864565\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,2]]},"references-count":57,"alternative-id":["10.3389\/frai.2026.1864565"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1864565","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,2]]},"article-number":"1864565"}}