{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T12:19:08Z","timestamp":1768220348733,"version":"3.49.0"},"reference-count":64,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T00:00:00Z","timestamp":1768176000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Explainability in deep learning (XDL) for healthcare is increasingly portrayed as essential for addressing the \u201cblack box\u201d problem in clinical artificial intelligence. However, this universal transparency mandate may create unintended consequences, including cognitive overload, spurious confidence, and workflow disruption. This paper examines a fundamental question: Is explainability a panacea that resolves AI\u2019s trust deficit, or a Pandora\u2019s box that introduces new risks? Drawing on general systems theory we demonstrate that the answer is profoundly context dependent. Through systemic analysis of current XDL methods, Saliency Maps, LIME, SHAP, and attention mechanisms, we reveal systematic disconnects between technical transparency and clinical utility. This paper argues that XDL is a context-dependent systemic property rather than a universal requirement. It functions as a panacea when proportionately applied to high-stakes reasoning tasks (cancer treatment planning, complex diagnosis) within integrated socio-technical architectures. Conversely, it becomes a Pandora\u2019s box when superficially imposed on routine operational functions (scheduling, preprocessing) or time-critical emergencies (e.g., cardiac arrest) where comprehensive explanation delays lifesaving intervention. The paper proposes a risk-stratified framework recognizing that a specific subset of healthcare AI applications\u2014those involving high-stakes clinical reasoning\u2014require comprehensive explainability, while other applications benefit from calibrated transparency appropriate to their clinical context. We conclude that explainability is neither a cure-all nor an inevitable harm, but rather a dynamic equilibrium requiring continuous rebalancing across technical, cognitive, and organizational dimensions.<\/jats:p>","DOI":"10.3390\/a19010063","type":"journal-article","created":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T09:13:01Z","timestamp":1768209181000},"page":"63","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Explainability in Deep Learning in Healthcare and Medicine: Panacea or Pandora\u2019s Box? A Systemic View"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9927-5343","authenticated-orcid":false,"given":"Wullianallur","family":"Raghupathi","sequence":"first","affiliation":[{"name":"Gabelli School of Business, Fordham University, 140 W. 62nd Street, New York, NY 10023, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,12]]},"reference":[{"key":"#cr-split#-ref_1.1","unstructured":"European Parliament and Council of the European Union (2024). Regulation"},{"key":"#cr-split#-ref_1.2","unstructured":"(EU) 2024\/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Off. J. Eur. Union, 1689, 1-144."},{"key":"ref_2","unstructured":"World Health Organization (2021). Ethics and Governance of Artificial Intelligence for Health: WHO Guidance, World Health Organization."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","article-title":"International evaluation of an AI system for breast cancer screening","volume":"577","author":"McKinney","year":"2020","journal-title":"Nature"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1038\/s41591-019-0508-1","article-title":"Clinical-grade computational pathology using weakly supervised deep learning on whole slide images","volume":"25","author":"Campanella","year":"2019","journal-title":"Nat. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","year":"2021","journal-title":"Lancet Digit. Health"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1038\/s41398-021-01224-x","article-title":"How machine-learning recommendations influence clinician treatment selections: The example of antidepressant selection","volume":"11","author":"Jacobs","year":"2021","journal-title":"Transl. Psychiatry"},{"key":"ref_8","unstructured":"Tonekaboni, S., Joshi, S., McCradden, M.D., and Goldenberg, A. (2019, January 8\u201310). What clinicians want: Contextualizing explainable machine learning for clinical end use. Proceedings of the Machine Learning for Healthcare Conference, Ann Arbor, MI, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1287\/mnsc.2.3.197","article-title":"General systems theory\u2014The skeleton of science","volume":"2","author":"Boulding","year":"1956","journal-title":"Manag. Sci."},{"key":"ref_10","first-page":"44","article-title":"Designing clinical decision support systems in health care: A systemic view","volume":"2","author":"Raghupathi","year":"2007","journal-title":"Int. J. Healthc. Inf. Syst. Inform."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/BF01060047","article-title":"Designing artificial intelligence applications in law: A systemic view","volume":"5","author":"Raghupathi","year":"1992","journal-title":"Syst. Pract."},{"key":"ref_12","first-page":"571","article-title":"Designing electronic health records versus total digital health systems","volume":"25","author":"Raghupathi","year":"2008","journal-title":"Syst. Res. Behav. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Singh, A., Sengupta, S., and Lakshminarayanan, V. (2020). Explainable deep learning models in medical image analysis. J. Imaging, 6.","DOI":"10.3390\/jimaging6060052"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Amann, J., Blasimme, A., Vayena, E., Frey, D., and Madai, V.I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med. Inform. Decis. Mak., 20.","DOI":"10.1186\/s12911-020-01332-6"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"107161","DOI":"10.1016\/j.cmpb.2022.107161","article-title":"Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011\u20132022)","volume":"226","author":"Loh","year":"2022","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Bellucci, M., Delestre, N., Malandain, N., and Zanni-Merk, C. (2021, January 8\u201310). Towards a terminology for a fully contextualized XAI. Proceedings of the International Conference on Knowledge-Based Intelligent Information & Engineering Systems, Szczecin, Poland.","DOI":"10.1016\/j.procs.2021.08.025"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e1548","DOI":"10.1002\/wsbm.1548","article-title":"Explainable deep learning in healthcare: A methodological survey from an attribution view","volume":"14","author":"Jin","year":"2022","journal-title":"WIREs Mech. Dis."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1038\/s41746-020-0254-2","article-title":"Machine intelligence in healthcare\u2014Perspectives on trustworthiness, explainability, usability, and transparency","volume":"3","author":"Cutillo","year":"2020","journal-title":"npj Digit. Med."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Glenn Cohen, I., Fernandez Lynch, H., Vayena, E., and Gasser, U. (2018). Medical malpractice and black-box medicine. Big Data, Health Law, and Bioethics, Cambridge University Press.","DOI":"10.1017\/9781108147972"},{"key":"ref_21","unstructured":"Chari, S., Chakraborty, P., Ghalwash, M., Seneviratne, O., Eyigoz, E.K., Gruen, D.M., Saiz, F.S., Chen, C.H., Rojas, P.M., and McGuinness, D.L. (2021). Leveraging Clinical Context for User-Centered Explainability: A Diabetes Use Case. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1197\/jamia.M1370","article-title":"Ten commandments for effective clinical decision support: Making the practice of evidence-based medicine a reality","volume":"10","author":"Bates","year":"2003","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kindermans, P.J., Hooker, S., Adebayo, J., Alber, M., Sch\u00fctt, K.T., D\u00e4hne, S., Erhan, D., and Kim, B. (2019). The (un)reliability of saliency methods. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer.","DOI":"10.1007\/978-3-030-28954-6_14"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1038\/s41591-018-0307-0","article-title":"The practical implementation of artificial intelligence technologies in medicine","volume":"25","author":"He","year":"2019","journal-title":"Nat. Med."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1337","DOI":"10.1038\/s41591-019-0548-6","article-title":"Do no harm: A roadmap for responsible machine learning for health care","volume":"25","author":"Wiens","year":"2019","journal-title":"Nat. Med."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable ai: A review of machine learning interpretability methods. Entropy, 23.","DOI":"10.3390\/e23010018"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: A comprehensive review","volume":"55","author":"Minh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1109\/JPROC.2021.3060483","article-title":"Explaining deep neural networks and beyond: A review of methods and applications","volume":"109","author":"Samek","year":"2021","journal-title":"Proc. IEEE"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery","volume":"16","author":"Lipton","year":"2018","journal-title":"Queue"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","article-title":"A survey on explainable artificial intelligence (xai): Toward medical xai","volume":"32","author":"Tjoa","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"103473","DOI":"10.1016\/j.artint.2021.103473","article-title":"What do we want from explainable artificial intelligence (XAI)?\u2014A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research","volume":"296","author":"Langer","year":"2021","journal-title":"Artif. Intell."},{"key":"ref_33","first-page":"4765","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"866","DOI":"10.7326\/M18-1990","article-title":"Ensuring fairness in machine learning to advance health equity","volume":"169","author":"Rajkomar","year":"2018","journal-title":"Ann. Intern. Med."},{"key":"ref_35","unstructured":"FDA (2022). Artificial Intelligence and Machine Learning (AI\/ML)-Enabled Medical Devices, U.S. Food and Drug Administration."},{"key":"ref_36","unstructured":"Kumar, I.E., Venkatasubramanian, S., Scheidegger, C., and Friedler, S. (2020, January 13\u201318). Problems with Shapley-value-based explanations as feature importance measures. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1038\/s42256-020-0197-y","article-title":"Causal inference and counterfactual prediction in machine learning for actionable healthcare","volume":"2","author":"Prosperi","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_38","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., and Eisenstein, J. (2018). Explainable prediction of medical codes from clinical text. arXiv.","DOI":"10.18653\/v1\/N18-1100"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021, January 3\u201310). On the dangers of stochastic parrots: Can language models be too big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual.","DOI":"10.1145\/3442188.3445922"},{"key":"ref_41","unstructured":"Jain, S., and Wallace, B.C. (2019). Attention is not explanation. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. (2019). Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv.","DOI":"10.18653\/v1\/P19-1580"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"842","DOI":"10.1162\/tacl_a_00349","article-title":"A primer in BERTology: What we know about how BERT works","volume":"8","author":"Rogers","year":"2021","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_44","unstructured":"Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6\u201311). Axiomatic attribution for deep networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Jacovi, A., and Goldberg, Y. (2020). Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.386"},{"key":"ref_46","unstructured":"von Bertalanffy, L. (1968). General System Theory: Foundations, Development, Applications, George Braziller."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/001872675100400101","article-title":"Some social and psychological consequences of the longwall method of coal-getting","volume":"4","author":"Trist","year":"1951","journal-title":"Hum. Relat."},{"key":"ref_48","unstructured":"Pasmore, W.A. (1988). Designing Effective Organizations: The Sociotechnical Systems Perspective, Wiley."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"i68","DOI":"10.1136\/qshc.2010.042085","article-title":"A new sociotechnical model for studying health information technology in complex adaptive healthcare systems","volume":"19","author":"Sittig","year":"2010","journal-title":"Qual. Saf. Health Care"},{"key":"ref_50","first-page":"3681","article-title":"Interpretation of neural networks is fragile","volume":"33","author":"Ghorbani","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_51","unstructured":"Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. (2018, January 3\u20138). Sanity checks for saliency maps. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., and Wortman Vaughan, J. (2020, January 25\u201330). Interpreting interpretability: Understanding data scientists\u2019 use of interpretability tools for machine learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3313831.3376219"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Oakden-Rayner, L., Dunnmon, J., Carneiro, G., and R\u00e9, C. (2020, January 2\u20134). Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. Proceedings of the ACM Conference on Health, Inference, and Learning, Toronto, ON, Canada.","DOI":"10.1145\/3368555.3384468"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should I trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_55","unstructured":"Alvarez-Melis, D., and Jaakkola, T.S. (2018). On the robustness of interpretability methods. arXiv."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020, January 7\u20138). Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods. Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, New York, NY, USA.","DOI":"10.1145\/3375627.3375830"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Molnar, C., Casalicchio, G., and Bischl, B. (2020, January 8\u201312). Interpretable machine learning\u2014A brief history, state-of-the-art and challenges. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Vilnius, Lithuania.","DOI":"10.1007\/978-3-030-65965-3_28"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Chen, H., Lundberg, S., and Lee, S.I. (2020). Explaining models by propagating Shapley values of local components. Explainable AI in Healthcare and Medicine: Building a Culture of Transparency and Accountability, Springer International Publishing.","DOI":"10.1007\/978-3-030-53352-6_24"},{"key":"ref_59","first-page":"841","article-title":"Counterfactual explanations without opening the black box: Automated decisions and the GDPR","volume":"31","author":"Wachter","year":"2017","journal-title":"Harv. J. Law Technol."},{"key":"ref_60","unstructured":"Verma, S., Dickerson, J., and Hines, K. (2020). Counterfactual explanations for machine learning: A review. arXiv."},{"key":"ref_61","unstructured":"Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. (2020, January 13\u201318). Concept bottleneck models. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_62","unstructured":"Pearl, J., and Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect, Basic Books."},{"key":"ref_63","unstructured":"Hern\u00e1n, M., and Robins, J.M. (2020). Causal Inference: What If, Chapman and Hall\/CRC."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/1\/63\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,12]],"date-time":"2026-01-12T09:20:46Z","timestamp":1768209646000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/1\/63"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,12]]},"references-count":64,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["a19010063"],"URL":"https:\/\/doi.org\/10.3390\/a19010063","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,12]]}}}