{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T17:17:24Z","timestamp":1776878244925,"version":"3.51.2"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T00:00:00Z","timestamp":1684713600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T00:00:00Z","timestamp":1684713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["187331"],"award-info":[{"award-number":["187331"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["52441_1"],"award-info":[{"award-number":["52441_1"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Explainable artificial intelligence (XAI) has emerged as a promising solution for addressing the implementation challenges of AI\/ML in healthcare. However, little is known about how developers and clinicians interpret XAI and what conflicting goals and requirements they may have. This paper presents the findings of a longitudinal multi-method study involving 112 developers and clinicians co-designing an XAI solution for a clinical decision support system. Our study identifies three key differences between developer and clinician mental models of XAI, including opposing goals (model interpretability vs. clinical plausibility), different sources of truth (data vs. patient), and the role of exploring new vs. exploiting old knowledge. Based on our findings, we propose design solutions that can help address the XAI conundrum in healthcare, including the use of causal inference models, personalized explanations, and ambidexterity between exploration and exploitation mindsets. Our study highlights the importance of considering the perspectives of both developers and clinicians in the design of XAI systems and provides practical recommendations for improving the effectiveness and usability of XAI in healthcare.<\/jats:p>","DOI":"10.1038\/s41746-023-00837-4","type":"journal-article","created":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T14:03:30Z","timestamp":1684764210000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":117,"title":["Solving the explainable AI conundrum by bridging clinicians\u2019 needs and developers\u2019 goals"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4200-8695","authenticated-orcid":false,"given":"Nadine","family":"Bienefeld","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8043-0961","authenticated-orcid":false,"given":"Jens Michael","family":"Boss","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0998-8847","authenticated-orcid":false,"given":"Rahel","family":"L\u00fcthy","sequence":"additional","affiliation":[]},{"given":"Dominique","family":"Brodbeck","sequence":"additional","affiliation":[]},{"given":"Jan","family":"Azzati","sequence":"additional","affiliation":[]},{"given":"Mirco","family":"Blaser","sequence":"additional","affiliation":[]},{"given":"Jan","family":"Willms","sequence":"additional","affiliation":[]},{"given":"Emanuela","family":"Keller","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,5,22]]},"reference":[{"key":"837_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00333-z","volume":"3","author":"B Mesk\u00f3","year":"2020","unstructured":"Mesk\u00f3, B. & G\u00f6r\u00f6g, M. A short guide for medical professionals in the era of artificial intelligence. Npj Digit. Med. 3, 1\u20138 (2020).","journal-title":"Npj Digit. Med."},{"key":"837_CR2","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1038\/s41591-021-01614-0","volume":"28","author":"P Rajpurkar","year":"2022","unstructured":"Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. AI in health and medicine. Nat. Med. 28, 31\u201338 (2022).","journal-title":"Nat. Med."},{"key":"837_CR3","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1167\/tvst.9.2.45","volume":"9","author":"RP Singh","year":"2020","unstructured":"Singh, R. P., Hom, G. L., Abramoff, M. D., Campbell, J. P. & Chiang, M. F. Current challenges and barriers to real-world artificial intelligence adoption for the healthcare system, provider, and the patient. Transl. Vis. Sci. Technol. 9, 45 (2020).","journal-title":"Transl. Vis. Sci. Technol."},{"key":"837_CR4","doi-asserted-by":"crossref","first-page":"205520762210744","DOI":"10.1177\/20552076221074488","volume":"8","author":"L Arbelaez Ossa","year":"2022","unstructured":"Arbelaez Ossa, L. et al. Re-focusing explainability in medicine. Digit. Health 8, 20552076221074490 (2022).","journal-title":"Digit. Health"},{"key":"837_CR5","doi-asserted-by":"publisher","first-page":"e0000016","DOI":"10.1371\/journal.pdig.0000016","volume":"1","author":"J Amann","year":"2022","unstructured":"Amann, J. et al. To explain or not to explain?\u2014Artificial intelligence explainability in clinical decision support systems. PLOS Digit. Health 1, e0000016 (2022).","journal-title":"PLOS Digit. Health"},{"key":"837_CR6","doi-asserted-by":"publisher","first-page":"1328","DOI":"10.1038\/s41591-021-01461-z","volume":"27","author":"S Kundu","year":"2021","unstructured":"Kundu, S. AI in medicine must be explainable. Nat. Med. 27, 1328\u20131328 (2021).","journal-title":"Nat. Med."},{"key":"837_CR7","doi-asserted-by":"publisher","first-page":"e1312","DOI":"10.1002\/widm.1312","volume":"9","author":"A Holzinger","year":"2019","unstructured":"Holzinger, A., Langs, G., Denk, H., Zatloukal, K. & M\u00fcller, H. Causability and explainability of artificial intelligence in medicine. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 9, e1312 (2019).","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"837_CR8","first-page":"e745","volume":"3","author":"M Ghassemi","year":"2021","unstructured":"Ghassemi, M., Oakden-Rayner, L. & Beam, A. L. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit. Health 3, e745\u2013e750 (2021).","journal-title":"Health"},{"key":"837_CR9","doi-asserted-by":"publisher","first-page":"e100495","DOI":"10.1136\/bmjhci-2021-100495","volume":"29","author":"D van de Sande","year":"2022","unstructured":"van de Sande, D. et al. Developing, implementing and governing artificial intelligence in medicine: a step-by-step approach to prevent an artificial intelligence winter. BMJ Health Care Inf. 29, e100495 (2022).","journal-title":"BMJ Health Care Inf."},{"key":"837_CR10","doi-asserted-by":"publisher","first-page":"330","DOI":"10.3102\/0034654312457429","volume":"82","author":"R Razzouk","year":"2012","unstructured":"Razzouk, R. & Shute, V. What Is Design Thinking and Why Is It Important? Rev. Educ. Res. 82, 330\u2013348 (2012).","journal-title":"Rev. Educ. Res."},{"key":"837_CR11","doi-asserted-by":"publisher","first-page":"425","DOI":"10.2307\/30036540","volume":"27","author":"V Venkatesh","year":"2003","unstructured":"Venkatesh, V., Morris, M. G., Davis, G. B. & Davis, F. D. User Acceptance of Information Technology: Toward a Unified View. MIS Q 27, 425 (2003).","journal-title":"MIS Q"},{"key":"837_CR12","unstructured":"Tabachnick, B. G., Fidell, L. S. & Ullman, J. B. Using multivariate statistics. (Pearson, 2019)."},{"key":"837_CR13","unstructured":"Field, A. Discovering statistics using IBM SPSS statistics. (Sage, 2019)."},{"key":"837_CR14","doi-asserted-by":"crossref","unstructured":"Glaser, B. G. & Strauss, A. L. Discovery of Grounded Theory: Strategies for Qualitative Research. (Routledge, 1967).","DOI":"10.1097\/00006199-196807000-00014"},{"key":"837_CR15","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1177\/1094428112452151","volume":"16","author":"DA Gioia","year":"2013","unstructured":"Gioia, D. A., Corley, K. G. & Hamilton, A. L. Seeking Qualitative Rigor in Inductive Research: Notes on the Gioia Methodology. Organ. Res. Methods 16, 15\u201331 (2013).","journal-title":"Organ. Res. Methods"},{"key":"837_CR16","doi-asserted-by":"publisher","unstructured":"Alsuradi, H., Park, W. & Eid, M. Explainable Classification of EEG Data for an Active Touch Task Using Shapley Values. In HCI International 2020 - Late Breaking Papers: Multimodality and Intelligence (eds. Stephanidis, C., Kurosu, M., Degen, H. & Reinerman-Jones, L.) 406\u2013416 (Springer International Publishing, 2020). https:\/\/doi.org\/10.1007\/978-3-030-60117-1_30.","DOI":"10.1007\/978-3-030-60117-1_30"},{"key":"837_CR17","unstructured":"Osmitrol (mannitol) dosing, indications, interactions, adverse effects, and more. https:\/\/reference.medscape.com\/drug\/osmitrol-mannitol-343061."},{"key":"837_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-0254-2","volume":"3","author":"CM Cutillo","year":"2020","unstructured":"Cutillo, C. M. et al. Machine intelligence in healthcare\u2014perspectives on trustworthiness, explainability, usability, and transparency. Npj Digit. Med. 3, 1\u20135 (2020).","journal-title":"Npj Digit. Med."},{"key":"837_CR19","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1038\/s41746-020-0288-5","volume":"3","author":"D Cirillo","year":"2020","unstructured":"Cirillo, D. et al. Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. Npj Digit. Med. 3, 81 (2020).","journal-title":"Npj Digit. Med."},{"key":"837_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-022-00695-6","volume":"5","author":"J Halamka","year":"2022","unstructured":"Halamka, J., Bydon, M., Cerrato, P. & Bhagra, A. Addressing racial disparities in surgical care with machine learning. Npj Digit. Med. 5, 1\u20134 (2022).","journal-title":"Npj Digit. Med."},{"key":"837_CR21","unstructured":"The Glasgow structured approach to assessment of the Glasgow Coma Scale. https:\/\/www.glasgowcomascale.org\/."},{"key":"837_CR22","doi-asserted-by":"publisher","first-page":"662","DOI":"10.1016\/j.bja.2018.04.024","volume":"121","author":"DW Tscholl","year":"2018","unstructured":"Tscholl, D. W. et al. Using an animated patient avatar to improve perception of vital sign information by anaesthesia professionals. Br. J. Anaesth. 121, 662\u2013671 (2018).","journal-title":"Br. J. Anaesth."},{"key":"837_CR23","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1109\/TVCG.2018.2865027","volume":"25","author":"BC Kwon","year":"2019","unstructured":"Kwon, B. C. et al. RetainVis: Visual Analytics with interpretable and interactive recurrent neural networks on electronic medical records. IEEE Trans. Vis. Comput. Graph. 25, 299\u2013309 (2019).","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"837_CR24","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1109\/TVCG.2021.3114836","volume":"28","author":"F Cheng","year":"2022","unstructured":"Cheng, F. et al. VBridge: Connecting the dots between features and data to explain healthcare models. IEEE Trans. Vis. Comput. Graph. 28, 378\u2013388 (2022).","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"837_CR25","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1038\/s42256-022-00445-z","volume":"4","author":"P Cui","year":"2022","unstructured":"Cui, P. & Athey, S. Stable learning establishes some common ground between causal inference and machine learning. Nat. Mach. Intell. 4, 110\u2013115 (2022).","journal-title":"Nat. Mach. Intell."},{"key":"837_CR26","doi-asserted-by":"publisher","first-page":"109","DOI":"10.5465\/19416521003691287","volume":"4","author":"D Lavie","year":"2010","unstructured":"Lavie, D., Stettner, U. & Tushman, M. L. Exploration and Exploitation Within and Across Organizations. Acad. Manag. Ann. 4, 109\u2013155 (2010).","journal-title":"Acad. Manag. Ann."},{"key":"837_CR27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-021-00554-w","volume":"5","author":"A Bartolome","year":"2022","unstructured":"Bartolome, A. & Prioleau, T. A computational framework for discovering digital biomarkers of glycemic control. Npj Digit. Med. 5, 1\u20139 (2022).","journal-title":"Npj Digit. Med."},{"key":"837_CR28","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1037\/0033-295X.94.2.211","volume":"94","author":"J Klayman","year":"1987","unstructured":"Klayman, J. & Ha, Y. Confirmation, disconfirmation, and information in hypothesis testing. Psychol. Rev. 94, 211\u2013228 (1987).","journal-title":"Psychol. Rev."},{"key":"837_CR29","unstructured":"Nickerson, R. S. Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. 46."},{"key":"837_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-022-00597-7","volume":"5","author":"KE Henry","year":"2022","unstructured":"Henry, K. E. et al. Human\u2013machine teaming is key to AI adoption: clinicians\u2019 experiences with a deployed machine learning system. Npj Digit. Med. 5, 1\u20136 (2022).","journal-title":"Npj Digit. Med."},{"key":"837_CR31","unstructured":"Tonekaboni, S., Joshi, S., McCradden, M. D. & Goldenberg, A. What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. in Proceedings of the 4th Machine Learning for Healthcare Conference 359\u2013380 (PMLR, 2019)."},{"key":"837_CR32","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1038\/s42256-020-0197-y","volume":"2","author":"M Prosperi","year":"2020","unstructured":"Prosperi, M. et al. Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nat. Mach. Intell. 2, 369\u2013375 (2020).","journal-title":"Nat. Mach. Intell."},{"key":"837_CR33","doi-asserted-by":"publisher","first-page":"1370","DOI":"10.1161\/STROKEAHA.120.032546","volume":"52","author":"M Megjhani","year":"2021","unstructured":"Megjhani, M. et al. Dynamic Detection of Delayed Cerebral Ischemia: A Study in 3 Centers. Stroke 52, 1370\u20131379 (2021).","journal-title":"Stroke"},{"key":"837_CR34","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s10994-006-6226-1","volume":"63","author":"P Geurts","year":"2006","unstructured":"Geurts, P., Ernst, D. & Wehenkel, L. Extremely randomized trees. Mach. Learn. 63, 3\u201342 (2006).","journal-title":"Mach. Learn."},{"key":"837_CR35","doi-asserted-by":"publisher","first-page":"1286","DOI":"10.1093\/jamia\/ocac064","volume":"29","author":"JM Boss","year":"2022","unstructured":"Boss, J. M. et al. ICU Cockpit: A platform for collecting multimodal waveform data, AI-based computational disease modeling and real-time decision support in the intensive care unit. J. Am. Med. Inform. Assoc. JAMIA 29, 1286\u20131291 (2022).","journal-title":"J. Am. Med. Inform. Assoc. JAMIA"},{"key":"837_CR36","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1016\/j.ijmedinf.2004.08.003","volume":"73","author":"MWM Jaspers","year":"2004","unstructured":"Jaspers, M. W. M., Steen, T., van den Bos, C. & Geenen, M. The think aloud method: A guide to user interface design. Int. J. Med. Inf. 73, 781\u2013795 (2004).","journal-title":"Int. J. Med. Inf."},{"key":"837_CR37","unstructured":"One platform to connect. Zoom https:\/\/zoom.us\/."},{"key":"837_CR38","doi-asserted-by":"publisher","first-page":"1245","DOI":"10.1097\/ACM.0000000000000388","volume":"89","author":"BC O\u2019Brien","year":"2014","unstructured":"O\u2019Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A. & Cook, D. A. Standards for Reporting Qualitative Research: A Synthesis of Recommendations. Acad. Med. 89, 1245\u20131251 (2014).","journal-title":"Acad. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00837-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00837-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00837-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T14:16:17Z","timestamp":1684764977000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00837-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,22]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["837"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00837-4","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-2326665\/v1","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,22]]},"assertion":[{"value":"29 November 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 May 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 May 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"94"}}