{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T13:37:12Z","timestamp":1780493832787,"version":"3.54.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,5,6]],"date-time":"2022-05-06T00:00:00Z","timestamp":1651795200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,6]],"date-time":"2022-05-06T00:00:00Z","timestamp":1651795200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ethics Inf Technol"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>There is a current debate about if, and in what sense, machine learning systems used in the medical context need to be explainable. Those arguing in favor contend these systems require post hoc explanations for each individual decision to increase trust and ensure accurate diagnoses. Those arguing against suggest the high accuracy and reliability of the systems is sufficient for providing epistemic justified beliefs without the need for explaining each individual decision. But, as we show, both solutions have limitations\u2014and it is unclear either address the epistemic worries of the medical professionals using these systems. We argue these systems do require an explanation, but an <jats:italic>institutional<\/jats:italic> explanation. These types of explanations provide the reasons why the medical professional should rely on the system in practice\u2014that is, they focus on trying to address the epistemic concerns of those using the system in specific contexts and specific occasions. But ensuring that these institutional explanations are fit for purpose means ensuring the institutions designing and deploying these systems are transparent about the assumptions baked into the system. This requires coordination with experts and end-users concerning how it will function in the field, the metrics used to evaluate its accuracy, and the procedures for auditing the system to prevent biases and failures from going unaddressed. We contend this broader explanation is necessary for either post hoc explanations or accuracy scores to be epistemically meaningful to the medical professional, making it possible for them to rely on these systems as effective and useful tools in their practices.<\/jats:p>","DOI":"10.1007\/s10676-022-09649-8","type":"journal-article","created":{"date-parts":[[2022,5,6]],"date-time":"2022-05-06T11:10:37Z","timestamp":1651835437000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Putting explainable AI in context: institutional explanations for medical AI"],"prefix":"10.1007","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3260-6248","authenticated-orcid":false,"given":"Mark","family":"Theunissen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"Browning","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,5,6]]},"reference":[{"key":"9649_CR1","volume-title":"Principles of biomedical ethics","author":"TL Beauchamp","year":"2019","unstructured":"Beauchamp, T. L., & Childress, J. F. (2019). Principles of biomedical ethics. Oxford University Press."},{"issue":"1","key":"9649_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00324-0","volume":"3","author":"S Benjamens","year":"2020","unstructured":"Benjamens, S., Dhunnoo, P., & Mesk\u00f3, B. (2020). The state of artificial intelligence-based FDA-approved medical devices and algorithms: An online database. NPJ Digital Medicine, 3(1), 1\u20138. https:\/\/doi.org\/10.1038\/s41746-020-00324-0","journal-title":"NPJ Digital Medicine"},{"issue":"4","key":"9649_CR3","first-page":"715","volume":"102","author":"PE Bickler","year":"2005","unstructured":"Bickler, P. E., Feiner, J. R., & Severinghaus, J. W. (2005). Effects of skin pigmentation on pulse oximeter accuracy at low saturation. The Journal of the American Society of Anesthesiologists, 102(4), 715\u2013719.","journal-title":"The Journal of the American Society of Anesthesiologists"},{"issue":"2","key":"9649_CR4","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1162\/POSC_a_00200","volume":"24","author":"J Biddle","year":"2016","unstructured":"Biddle, J. (2016). Inductive risk, epistemic risk, and overdiagnosis of disease. Perspectives on Science, 24(2), 192\u2013205. https:\/\/doi.org\/10.1162\/POSC_a_00200","journal-title":"Perspectives on Science"},{"key":"9649_CR5","doi-asserted-by":"publisher","first-page":"101200","DOI":"10.1016\/j.shpsc.2019.101200","volume":"79","author":"J Biddle","year":"2020","unstructured":"Biddle, J. (2020). Epistemic risks in cancer screening: Implications for ethics and policy. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 79, 101200. https:\/\/doi.org\/10.1016\/j.shpsc.2019.101200","journal-title":"Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences"},{"key":"9649_CR6","first-page":"215","volume-title":"Exploring inductive risk: Case studies of values in science","author":"JB Biddle","year":"2017","unstructured":"Biddle, J. B., & Kukla, R. (2017). The geography of epistemic risk. In K. C. Elliott & T. Richards (Eds.), Exploring inductive risk: Case studies of values in science (pp. 215\u2013237). Oxford University Press."},{"key":"9649_CR7","doi-asserted-by":"publisher","DOI":"10.1177\/2053951715622512","author":"J Burrell","year":"2016","unstructured":"Burrell, J. (2016). How the machine \u2018thinks\u2019: Understanding opacity in machine learning algorithms. Big Data & Society. https:\/\/doi.org\/10.1177\/2053951715622512","journal-title":"Big Data & Society"},{"key":"9649_CR8","doi-asserted-by":"crossref","unstructured":"Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1721\u20131730).","DOI":"10.1145\/2783258.2788613"},{"issue":"4","key":"9649_CR9","doi-asserted-by":"publisher","first-page":"568","DOI":"10.1086\/709729","volume":"87","author":"KA Creel","year":"2020","unstructured":"Creel, K. A. (2020). Transparency in complex computational systems. Philosophy of Science, 87(4), 568\u2013589.","journal-title":"Philosophy of Science"},{"key":"9649_CR10","doi-asserted-by":"publisher","DOI":"10.1007\/s11229-020-02773-2","author":"R Dotan","year":"2020","unstructured":"Dotan, R. (2020). Theory choice, non-epistemic values, and machine learning. Synthese. https:\/\/doi.org\/10.1007\/s11229-020-02773-2","journal-title":"Synthese"},{"key":"9649_CR11","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103498","volume":"297","author":"JM Dur\u00e1n","year":"2021","unstructured":"Dur\u00e1n, J. M. (2021). Dissecting scientific explanation in AI (sXAI): A case for medicine and healthcare. Artificial Intelligence, 297, 103498. https:\/\/doi.org\/10.1016\/j.artint.2021.103498","journal-title":"Artificial Intelligence"},{"issue":"4","key":"9649_CR12","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1007\/s11023-018-9481-6","volume":"28","author":"JM Dur\u00e1n","year":"2018","unstructured":"Dur\u00e1n, J. M., & Formanek, N. (2018). Grounds for trust: Essential epistemic opacity and computational reliabilism. Minds and Machines, 28(4), 645\u2013666. https:\/\/doi.org\/10.1007\/s11023-018-9481-6","journal-title":"Minds and Machines"},{"issue":"5","key":"9649_CR13","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1136\/medethics-2020-106820","volume":"47","author":"JM Dur\u00e1n","year":"2021","unstructured":"Dur\u00e1n, J. M., & Jongsma, K. R. (2021). Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI. Journal of Medical Ethics, 47(5), 329\u2013335. https:\/\/doi.org\/10.1136\/medethics-2020-106820","journal-title":"Journal of Medical Ethics"},{"issue":"7","key":"9649_CR14","doi-asserted-by":"publisher","first-page":"e184","DOI":"10.1080\/01421590802144260","volume":"30","author":"PJH Engel","year":"2008","unstructured":"Engel, P. J. H. (2008). Tacit knowledge and visual expertise in medical diagnostic reasoning: Implications for medical education. Medical Teacher, 30(7), e184\u2013e188.","journal-title":"Medical Teacher"},{"key":"9649_CR15","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A., Kuprel, B., Novoa, R., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115\u2013118. https:\/\/doi.org\/10.1038\/nature21056","journal-title":"Nature"},{"key":"9649_CR16","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/7585.001.0001","volume-title":"Value sensitive design: Shaping technology with moral imagination","author":"B Friedman","year":"2019","unstructured":"Friedman, B., & Hendry, D. G. (2019). Value sensitive design: Shaping technology with moral imagination. MIT Press."},{"key":"9649_CR17","unstructured":"Garcia de Jes\u00fas, E. (2021). People with rare blood clots after a COVID-19 jab share an uncommon immune response. Retrieved from https:\/\/www.sciencenews.org\/article\/covid-vaccine-blood-clot-immune-astrazeneca-johnson-johnson"},{"issue":"31","key":"9649_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00385-9","volume":"4","author":"S Gaube","year":"2021","unstructured":"Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021). Do as AI say: Susceptibility in deployment of clinical decision-aids. NPJ Digital Medicine, 4(31), 1\u20138. https:\/\/doi.org\/10.1038\/s41746-021-00385-9","journal-title":"NPJ Digital Medicine"},{"key":"9649_CR19","doi-asserted-by":"publisher","DOI":"10.5195\/philmed.2021.27","author":"K Genin","year":"2021","unstructured":"Genin, K., & Grote, T. (2021). Randomized controlled trials in medical AI: A methodological critique. Philosophy of Medicine. https:\/\/doi.org\/10.5195\/philmed.2021.27","journal-title":"Philosophy of Medicine"},{"issue":"11","key":"9649_CR20","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","volume":"3","author":"M Ghassemi","year":"2021","unstructured":"Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021). The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(11), e745\u2013e750. https:\/\/doi.org\/10.1016\/S2589-7500(21)00208-9","journal-title":"The Lancet Digital Health"},{"issue":"3","key":"9649_CR21","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1136\/medethics-2019-105586","volume":"46","author":"T Grote","year":"2020","unstructured":"Grote, T., & Berens, P. (2020). On the ethics of algorithmic decision-making in healthcare. Journal of Medical Ethics, 46(3), 205\u2013211. https:\/\/doi.org\/10.1136\/medethics-2019-105586","journal-title":"Journal of Medical Ethics"},{"key":"9649_CR22","unstructured":"Heaven, W. D. (2020). Google\u2019s medical AI was super accurate in a lab. Real life was a different story. Retrieved October 22, 2021, from https:\/\/www.technologyreview.com\/2020\/04\/27\/1000658\/google-medical-ai-accurate-lab-real-life-clinic-covid-diabetes-retina-disease\/"},{"key":"9649_CR23","unstructured":"Heaven, W. D. (2021). Hundreds of AI tools have been built to catch covid. None of them helped. MIT Technology Review. Retrieved October 6, 2021, from https:\/\/www.technologyreview.com\/2021\/07\/30\/1030329\/machine-learning-ai-failed-covid-hospital-diagnosis-pandemic\/"},{"key":"9649_CR24","unstructured":"Holzinger, A., Biemann, C., Pattichis, C., & Kell, D. (2017). What do we need to build explainable AI systems for the medical domain? https:\/\/arxiv.org\/1712.09923"},{"key":"9649_CR25","doi-asserted-by":"publisher","DOI":"10.1007\/s11229-020-02696-y","author":"GM Johnson","year":"2020","unstructured":"Johnson, G. M. (2020). Algorithmic bias: On the implicit biases of social technology. Synthese. https:\/\/doi.org\/10.1007\/s11229-020-02696-y","journal-title":"Synthese"},{"key":"9649_CR26","unstructured":"Khetpal, V., & Shah, N. (2021). How a largely untested AI algorithm crept into hundred of hospitals. Retrieved June 17, 2021, from https:\/\/www.fastcompany.com\/90641343\/epic-deterioration-index-algorithm-pandemic-concerns."},{"issue":"3","key":"9649_CR27","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton, Z. C. (2018). The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3), 31\u201357.","journal-title":"Queue"},{"issue":"1","key":"9649_CR28","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1002\/hast.973","volume":"49","author":"AJ London","year":"2019","unstructured":"London, A. J. (2019). Artificial intelligence and black-box medical decisions: Accuracy versus explainability. Hastings Center Report, 49(1), 15\u201321. https:\/\/doi.org\/10.1002\/hast.973","journal-title":"Hastings Center Report"},{"key":"9649_CR100","doi-asserted-by":"publisher","unstructured":"Nissenbaum, H. (2011). A contextual approach to privacy online. Daedalus, 140(4), 32\u201348. https:\/\/doi.org\/10.1162\/DAED_a_00113.","DOI":"10.1162\/DAED_a_00113"},{"key":"9649_CR29","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-022-09632-3","author":"R Nyrup","year":"2022","unstructured":"Nyrup, R., & Robinson, D. (2022). Explanatory pragmatism: A context-sensitive framework for explainable medical AI. Ethics and Information Technology. https:\/\/doi.org\/10.1007\/s10676-022-09632-3","journal-title":"Ethics and Information Technology"},{"key":"9649_CR30","volume-title":"Personal knowledge","author":"M Polanyi","year":"1958","unstructured":"Polanyi, M. (1958). Personal knowledge. University of Chicago Press."},{"key":"9649_CR31","first-page":"66","volume":"33","author":"WN Price II","year":"2019","unstructured":"Price, W. N., II. (2019). Medical AI and Contextual Bias. Harvard Journal of Law and Technology., 33, 66.","journal-title":"Harvard Journal of Law and Technology."},{"issue":"5","key":"9649_CR32","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","volume":"1","author":"C Rudin","year":"2019","unstructured":"Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206\u2013215. https:\/\/doi.org\/10.1038\/s42256-019-0048-x","journal-title":"Nature Machine Intelligence"},{"issue":"2","key":"9649_CR33","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1111\/bioe.12887","volume":"36","author":"M Sand","year":"2022","unstructured":"Sand, M., Dur\u00e1n, J. M., & Jongsma, K. R. (2022). Responsibility beyond design: Physicians\u2019 requirements for ethical medical AI. Bioethics, 36(2), 162\u2013169. https:\/\/doi.org\/10.1111\/bioe.12887","journal-title":"Bioethics"},{"key":"9649_CR34","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1038\/s41746-019-0106-0","volume":"2","author":"S Sarwar","year":"2019","unstructured":"Sarwar, S., Dent, A., Faust, K., Richer, M., Djuric, U., Van Ommeren, R., & Diamandis, P. (2019). Physician perspectives on integration of artificial intelligence into diagnostic pathology. NPJ Digital Medicine, 2, 28. https:\/\/doi.org\/10.1038\/s41746-019-0106-0","journal-title":"NPJ Digital Medicine"},{"key":"9649_CR35","doi-asserted-by":"publisher","DOI":"10.1513\/AnnalsATS.202006-698OC","author":"K Singh","year":"2020","unstructured":"Singh, K., Valley, T. S., Tang, S., Li, B. Y., Kamran, F., Sjoding, M. W., Wiens, J., Otles, E., Donnelly, J. P., Wei, M. Y., McBride, J. P., Cao, J., Penoza, C., Ayanian, J. Z., & Nallamothu, B. K. (2020). Evaluating a widely implemented proprietary deterioration index model among hospitalized covid-19 patients. Annals of the American Thoracic Society. https:\/\/doi.org\/10.1513\/AnnalsATS.202006-698OC","journal-title":"Annals of the American Thoracic Society"},{"issue":"25","key":"9649_CR36","doi-asserted-by":"publisher","first-page":"2477","DOI":"10.1056\/NEJMc2029240","volume":"383","author":"MW Sjoding","year":"2020","unstructured":"Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020). Racial bias in pulse oximetry measurement. New England Journal of Medicine, 383(25), 2477\u20132478.","journal-title":"New England Journal of Medicine"},{"key":"9649_CR37","volume-title":"How FDA regulates artificial intelligence in medical products","author":"Staff","year":"2021","unstructured":"Staff. (2021). How FDA regulates artificial intelligence in medical products. Pew Charitable Trusts."},{"key":"9649_CR38","doi-asserted-by":"publisher","DOI":"10.1093\/bjps\/axz035","author":"E Sullivan","year":"2019","unstructured":"Sullivan, E. (2019). Understanding from machine learning models. The British Journal for the Philosophy of Science. https:\/\/doi.org\/10.1093\/bjps\/axz035","journal-title":"The British Journal for the Philosophy of Science"},{"key":"9649_CR39","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-021-00038-3","author":"S Umbrello","year":"2021","unstructured":"Umbrello, S., & van de Poel, I. (2021). Mapping value sensitive design onto AI for social good principles. AI and Ethics. https:\/\/doi.org\/10.1007\/s43681-021-00038-3","journal-title":"AI and Ethics"},{"key":"9649_CR40","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1007\/s13347-019-00382-7","volume":"34","author":"C Zednik","year":"2021","unstructured":"Zednik, C. (2021). Solving the black box problem: A normative framework for explainable artificial intelligence. Philosophy and Technology, 34, 265\u2013288. https:\/\/doi.org\/10.1007\/s13347-019-00382-7","journal-title":"Philosophy and Technology"},{"key":"9649_CR41","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1007\/s13347-018-0330-6","volume":"32","author":"J Zerilli","year":"2019","unstructured":"Zerilli, J., Knott, A., Maclaurin, J., & Gavaghan, C. (2019). (2019) Transparency in algorithmic and human decision-making: Is there a double standard? Philosophy and Technology, 32, 661\u2013683. https:\/\/doi.org\/10.1007\/s13347-018-0330-6","journal-title":"Philosophy and Technology"}],"container-title":["Ethics and Information Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10676-022-09649-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10676-022-09649-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10676-022-09649-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T04:29:55Z","timestamp":1656476995000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10676-022-09649-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,6]]},"references-count":42,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["9649"],"URL":"https:\/\/doi.org\/10.1007\/s10676-022-09649-8","relation":{},"ISSN":["1388-1957","1572-8439"],"issn-type":[{"value":"1388-1957","type":"print"},{"value":"1572-8439","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,6]]},"assertion":[{"value":"11 April 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"23"}}