{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T11:31:00Z","timestamp":1774611060985,"version":"3.50.1"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Johannes Kepler University Linz"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["AI Ethics"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In the past few years, machine learning (ML) tools have been implemented with success in the medical context. However, several practitioners have raised concerns about the lack of transparency\u2014at the algorithmic level\u2014of many of these tools; and solutions from the field of explainable AI (XAI) have been seen as a way to open the \u2018black box\u2019 and make the tools more trustworthy. Recently, Alex London has argued that in the medical context we do not need machine learning tools to be interpretable at the algorithmic level to make them trustworthy, as long as they meet some strict empirical desiderata. In this paper, we analyse and develop London\u2019s position. In particular, we make two claims. First, we claim that London\u2019s solution to the problem of trust can potentially address another problem, which is how to evaluate the reliability of ML tools in medicine for regulatory purposes. Second, we claim that to deal with this problem, we need to develop London\u2019s views by shifting the focus from the opacity of algorithmic details to the opacity of the way in which ML tools are trained and built. We claim that to regulate AI tools and evaluate their reliability, agencies need an explanation of how ML tools have been built, which requires documenting and justifying the technical choices that practitioners have made in designing such tools. This is because different algorithmic designs may lead to different outcomes, and to the realization of different purposes. However, given that technical choices underlying algorithmic design are shaped by value-laden considerations, opening the black box of the design process means also making transparent and\u00a0motivating (technical and ethical) values and preferences behind such choices. Using tools from philosophy of technology and philosophy of science, we elaborate a framework showing how an explanation of the training processes of ML tools in medicine should look like.<\/jats:p>","DOI":"10.1007\/s43681-022-00141-z","type":"journal-article","created":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T18:05:11Z","timestamp":1644948311000},"page":"801-814","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":51,"title":["Explainable machine learning practices: opening another black box for reliable medical AI"],"prefix":"10.1007","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1409-8240","authenticated-orcid":false,"given":"Emanuele","family":"Ratti","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Graves","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,2,15]]},"reference":[{"issue":"4","key":"141_CR1","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1007\/s10278-017-9984-3","volume":"30","author":"Z Akkus","year":"2017","unstructured":"Akkus, Z., Ali, I., Sedl\u00e1\u0159, J., Agrawal, J.P., Parney, I.F., Giannini, C., Erickson, B.J.: Predicting deletion of chromosomal arms 1p\/19q in low-grade gliomas from MR images using machine intelligence. J. Digit. Imaging 30(4), 469\u2013476 (2017). https:\/\/doi.org\/10.1007\/s10278-017-9984-3","journal-title":"J. Digit. Imaging"},{"issue":"4","key":"141_CR2","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.1177\/00018392211016755","volume":"66","author":"C Anthony","year":"2021","unstructured":"Anthony, C.: When knowledge work and analytical technologies collide: the practices and consequences of black boxing algorithmic technologies. Adm. Sci. Q. 66(4), 1173\u20131212 (2021). https:\/\/doi.org\/10.1177\/00018392211016755","journal-title":"Adm. Sci. Q."},{"key":"141_CR3","doi-asserted-by":"crossref","unstructured":"Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., Bao, M.: The Values Encoded in Machine Learning Research. arXiv:2106.15590 (2021)","DOI":"10.1145\/3531146.3533083"},{"key":"141_CR4","doi-asserted-by":"publisher","unstructured":"Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N.: Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015-August, 1721\u20131730 (2015). https:\/\/doi.org\/10.1145\/2783258.2788613","DOI":"10.1145\/2783258.2788613"},{"issue":"5","key":"141_CR5","doi-asserted-by":"publisher","first-page":"410","DOI":"10.1038\/s41563-019-0345-0","volume":"18","author":"C Chen","year":"2019","unstructured":"Chen, C., Liu, Y., Peng, L.: How to develop machine learning models for healthcare. Nat. Mater. 18(5), 410\u2013414 (2019). https:\/\/doi.org\/10.1038\/s41563-019-0345-0","journal-title":"Nat. Mater."},{"issue":"12","key":"141_CR6","doi-asserted-by":"publisher","first-page":"1415","DOI":"10.1016\/j.jacr.2016.07.010","volume":"13","author":"K Chockley","year":"2016","unstructured":"Chockley, K., Emanuel, E.: The end of radiology? Three threats to the future practice of radiology. J. Am. Coll. Radiol. 13(12), 1415\u20131420 (2016). https:\/\/doi.org\/10.1016\/j.jacr.2016.07.010","journal-title":"J. Am. Coll. Radiol."},{"issue":"20","key":"141_CR7","doi-asserted-by":"publisher","first-page":"741","DOI":"10.2307\/2024640","volume":"72","author":"R Cummins","year":"1975","unstructured":"Cummins, R.: Functional analysis. J. Philos. 72(20), 741\u2013765 (1975)","journal-title":"J. Philos."},{"key":"141_CR8","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226039824.001.0001","volume-title":"In Search of Mechanisms","author":"C Craver","year":"2013","unstructured":"Craver, C., Darden, L.: In Search of Mechanisms. The University of Chicago Press, Chicago (2013)"},{"issue":"05","key":"141_CR9","doi-asserted-by":"publisher","first-page":"7659","DOI":"10.1609\/aaai.v34i05.6267","volume":"34","author":"S Dev","year":"2020","unstructured":"Dev, S., Li, T., Phillips, J.M., Srikumar, V.: On measuring and mitigating biased inferences of word embeddings. Proc. AAAI Conf. Artif. Intell. 34(05), 7659\u20137666 (2020). https:\/\/doi.org\/10.1609\/aaai.v34i05.6267","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"issue":"4","key":"141_CR10","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1093\/jamia\/ocz229","volume":"27","author":"WK Diprose","year":"2020","unstructured":"Diprose, W.K., Buist, N., et al.: Physician understanding, explainability, and trust in a hypothetical machine learning risk calculator. J. Am. Med. Inform. Assoc. 27(4), 592\u2013600 (2020). https:\/\/doi.org\/10.1093\/jamia\/ocz229","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"141_CR11","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctt6wrc78","volume-title":"Science, Policy, and the Value-Free Ideal","author":"H Douglas","year":"2009","unstructured":"Douglas, H.: Science, Policy, and the Value-Free Ideal. University of Pittsburgh Press, Pittsburgh (2009)"},{"key":"141_CR12","volume-title":"Exploring Inductive Risk\u2014Case Studies of Values and Science","year":"2017","unstructured":"Elliott, K., Ted, R. (eds.): Exploring Inductive Risk\u2014Case Studies of Values and Science. Oxford University Press, Oxford (2017)"},{"key":"141_CR13","volume-title":"A Critical Reflection on Automated Science\u2014Will Science Remain Human?","author":"Emanuele Ratti","year":"2020","unstructured":"Emanuele, R.: Phronesis and automated science: the case of machine learning and biology. In: Bertolaso, M., Sterpetti, F. (eds.) A Critical Reflection on Automated Science\u2014Will Science Remain Human? Springer, Berlin (2020)"},{"key":"141_CR14","first-page":"1","volume-title":"Proposed Regulatory Framework for Modifications to Artificial Intelligence\/Machine Learning (AI\/ML)-Based Software as a Medical Device (SaMD)-Discussion Paper and Request for Feedback","author":"FDA","year":"2019","unstructured":"FDA: Proposed Regulatory Framework for Modifications to Artificial Intelligence\/Machine Learning (AI\/ML)-Based Software as a Medical Device (SaMD)-Discussion Paper and Request for Feedback, pp. 1\u201320. U.S Food & Drug Administration, New York (2019)"},{"key":"141_CR15","unstructured":"Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., Wallach, H., Daum\u00e9 III, H., Crawford, K.: Datasheets for Datasets. arXiv:1803.09010 (arXiv preprint) (2018)"},{"issue":"11","key":"141_CR16","doi-asserted-by":"publisher","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","volume":"178","author":"MA Gianfrancesco","year":"2018","unstructured":"Gianfrancesco, M.A., Tamang, S., Yazdany, J., Schmajuk, G.: Potential biases in machine learning algorithms using electronic health record data. JAMA Intern. Med. 178(11), 1544 (2018). https:\/\/doi.org\/10.1001\/jamainternmed.2018.3763","journal-title":"JAMA Intern. Med."},{"issue":"10","key":"141_CR17","doi-asserted-by":"publisher","first-page":"1122","DOI":"10.1038\/s41592-021-01205-4","volume":"18","author":"B Heil","year":"2021","unstructured":"Heil, B., Hoffman, M., Markowetz, F., Lee, S.-I., Greene, C., Hicks, S.: Reproducibility standards for machine learning in the life sciences. Nat. Methods 18(10), 1122\u20131127 (2021). https:\/\/doi.org\/10.1038\/s41592-021-01205-4","journal-title":"Nat. Methods"},{"key":"141_CR18","volume-title":"Philosophy of Natural Science","author":"C Hempel","year":"1966","unstructured":"Hempel, C.: Philosophy of Natural Science. Prentice-Hall, New York (1966)"},{"key":"141_CR19","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/s13218-020-00636-z","volume-title":"\u201cMeasuring the Quality of Explanations: The System Causability Scale (SCS): Comparing Human and Machine Explanations\u201d KI-Kunstliche Intelligenz 34 (2)","author":"A Holzinger","year":"2020","unstructured":"Holzinger, A., Carrington, A., M\u00fcller, H.: \u201cMeasuring the Quality of Explanations: The System Causability Scale (SCS): Comparing Human and Machine Explanations\u201d KI-Kunstliche Intelligenz 34 (2), pp. 193\u2013198. Springer, Berlin (2020). https:\/\/doi.org\/10.1007\/s13218-020-00636-z"},{"issue":"7704","key":"141_CR20","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1038\/d41586-018-05049-5","volume":"557","author":"TC Knepper","year":"2018","unstructured":"Knepper, T.C., McLeod, H.L.: When will clinical trials finally reflect diversity? Nature 557(7704), 157\u2013159 (2018). https:\/\/doi.org\/10.1038\/d41586-018-05049-5","journal-title":"Nature"},{"key":"141_CR21","first-page":"20","volume":"20","author":"JA Kroll","year":"2018","unstructured":"Kroll, J.A.: The fallacy of inscrutability. Philos. Trans. R. Soc. A 20, 20 (2018)","journal-title":"Philos. Trans. R. Soc. A"},{"key":"141_CR22","doi-asserted-by":"publisher","first-page":"320","DOI":"10.7208\/chicago\/9780226217239.001.0001","volume-title":"The Essential Tension","author":"T Kuhn","year":"1977","unstructured":"Kuhn, T.: Rationality, value judgment, and theory choice. In: The Essential Tension, pp. 320\u2013339. Chicago University Press, Chicago (1977)"},{"key":"141_CR23","unstructured":"Lehr, D., Ohm, P.: Playing with the Data: What Legal Scholars Should Learn About Machine Learning (2017)."},{"issue":"1","key":"141_CR24","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1002\/hast.973","volume":"49","author":"AJ London","year":"2019","unstructured":"London, A.J.: Artificial intelligence and black-box medical decisions: accuracy versus explainability. Hastings Cent. Rep. 49(1), 15\u201321 (2019). https:\/\/doi.org\/10.1002\/hast.973","journal-title":"Hastings Cent. Rep."},{"key":"141_CR25","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-020-09564-w","author":"M Loi","year":"2020","unstructured":"Loi, M., Ferrario, A., Vigan\u00f2, E.: Transparency as design publicity: explaining and justifying inscrutable algorithms. Ethics Inf. Technol. (2020). https:\/\/doi.org\/10.1007\/s10676-020-09564-w","journal-title":"Ethics Inf. Technol."},{"key":"141_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/2053951717700925","volume":"4","author":"I Lowrie","year":"2017","unstructured":"Lowrie, I.: Algorithmic rationality: epistemology and efficiency in the data sciences. Big Data Soc. 4, 1 (2017). https:\/\/doi.org\/10.1177\/2053951717700925","journal-title":"Big Data Soc."},{"key":"141_CR27","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2019.2962680","author":"F Mart\u00ednez-Plumed","year":"2019","unstructured":"Mart\u00ednez-Plumed, F., Contreras-Ochando, L., Ferri, C., Hern\u00e1ndez Orallo, J., Kull, M., Lachiche, N., Ram\u00edrez Quintana, M.J., Flach, P.A.: CRISP-DM twenty years later: from data mining processes to data science trajectories. IEEE Trans. Knowl. Data Eng. (2019). https:\/\/doi.org\/10.1109\/TKDE.2019.2962680","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"141_CR28","first-page":"686","volume":"2","author":"E Mcmullin","year":"1983","unstructured":"Mcmullin, E.: Values in science. Proc. Biennial Meet. Philos. Sci. Assoc. 2, 686\u2013709 (1983)","journal-title":"Proc. Biennial Meet. Philos. Sci. Assoc."},{"key":"141_CR29","doi-asserted-by":"crossref","unstructured":"Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model cards for model reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 220\u2013229 (2019)","DOI":"10.1145\/3287560.3287596"},{"key":"141_CR30","unstructured":"Mulligan, D.K., Kluttz, D.N., Kohli, N.: Shaping Our Tools: Contestability as a Means to Promote Responsible Algorithmic Decision Making in the Professions. https:\/\/ssrn.com\/abstract=3311894 (2019)"},{"issue":"1","key":"141_CR31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1086\/287231","volume":"20","author":"R Rudner","year":"1953","unstructured":"Rudner, R.: The scientist qua scientist makes value judgement. Philos. Sci. 20(1), 1\u20136 (1953)","journal-title":"Philos. Sci."},{"issue":"3","key":"141_CR32","doi-asserted-by":"publisher","first-page":"1085","DOI":"10.2139\/ssrn.3126971","volume":"87","author":"AD Selbst","year":"2018","unstructured":"Selbst, A.D., Barocas, S.: The intuitive appeal of explainable machines. Fordham Law Rev. 87(3), 1085\u20131139 (2018). https:\/\/doi.org\/10.2139\/ssrn.3126971","journal-title":"Fordham Law Rev."},{"issue":"21","key":"141_CR33","doi-asserted-by":"publisher","first-page":"2199","DOI":"10.1001\/jama.2018.17163","volume":"320","author":"EH Shortliffe","year":"2018","unstructured":"Shortliffe, E.H., Sep\u00falveda, M.J.: Clinical decision support in the era of artificial intelligence. J. Am. Med. Assoc. 320(21), 2199\u20132200 (2018). https:\/\/doi.org\/10.1001\/jama.2018.17163","journal-title":"J. Am. Med. Assoc."},{"key":"141_CR34","volume-title":"Deep Medicine\u2014How Artificial Intelligence Can Make Healthcare Human Again","author":"EJ Topol","year":"2019","unstructured":"Topol, E.J.: Deep Medicine\u2014How Artificial Intelligence Can Make Healthcare Human Again. Basic Books, New York (2019)"},{"issue":"3","key":"141_CR35","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1007\/s11023-020-09537-4","volume":"30","author":"I van de Poel","year":"2020","unstructured":"van de Poel, I.: Embedding values in artificial intelligence (AI) systems. Mind. Mach. 30(3), 385\u2013409 (2020). https:\/\/doi.org\/10.1007\/s11023-020-09537-4","journal-title":"Mind. Mach."},{"issue":"11\u201312","key":"141_CR36","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1080\/09544828.2011.603692","volume":"22","author":"D van Eck","year":"2011","unstructured":"van Eck, D.: Supporting design knowledge exchange by converting models of functional decomposition. J. Eng. Des. 22(11\u201312), 839\u2013858 (2011). https:\/\/doi.org\/10.1080\/09544828.2011.603692","journal-title":"J. Eng. Des."},{"issue":"3","key":"141_CR37","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1007\/s13194-015-0111-3","volume":"5","author":"D van Eck","year":"2015","unstructured":"van Eck, D.: Mechanistic explanation in engineering science. Eur. J. Philos. Sci. 5(3), 349\u2013375 (2015). https:\/\/doi.org\/10.1007\/s13194-015-0111-3","journal-title":"Eur. J. Philos. Sci."},{"issue":"18","key":"141_CR38","doi-asserted-by":"publisher","first-page":"1806","DOI":"10.1001\/jama.2019.16489","volume":"322","author":"L Yun","year":"2019","unstructured":"Yun, L., Chen, C., et al.: How to read articles that use machine learning: users\u2019 guides to the medical literature. J. Am. Med. Assoc. 322(18), 1806\u20131816 (2019). https:\/\/doi.org\/10.1001\/jama.2019.16489","journal-title":"J. Am. Med. Assoc."},{"issue":"4","key":"141_CR39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0231166","volume":"15","author":"E Zihni","year":"2020","unstructured":"Zihni, E., Madai, V.I., et al.: Opening the black box of artificial intelligence for clinical decision support: a study predicting stroke outcome. PLoS\u00a0One 15(4), 1\u201315 (2020). https:\/\/doi.org\/10.1371\/journal.pone.0231166","journal-title":"PLoS\u00a0One"}],"container-title":["AI and Ethics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-022-00141-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s43681-022-00141-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-022-00141-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T08:02:55Z","timestamp":1667376175000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s43681-022-00141-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,15]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["141"],"URL":"https:\/\/doi.org\/10.1007\/s43681-022-00141-z","relation":{},"ISSN":["2730-5953","2730-5961"],"issn-type":[{"value":"2730-5953","type":"print"},{"value":"2730-5961","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,15]]},"assertion":[{"value":"16 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 February 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}