{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T14:03:44Z","timestamp":1781273024470,"version":"3.54.1"},"reference-count":112,"publisher":"Association for Computing Machinery (ACM)","issue":"10","license":[{"start":{"date-parts":[[2025,5,7]],"date-time":"2025-05-07T00:00:00Z","timestamp":1746576000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Future Artificial Intelligence Research","award":["PE0000013 - CUP B53C22003630006"],"award-info":[{"award-number":["PE0000013 - CUP B53C22003630006"]}]},{"name":"Italian Ministry of University and Research"},{"name":"Digital-Driven Diagnostics, Prognostics, and Therapeutics for Sustainable Health Care","award":["PNC0000001 - CUP B83C22006120001"],"award-info":[{"award-number":["PNC0000001 - CUP B83C22006120001"]}]},{"name":"European Union -NextGenerationEU"},{"name":"Digital Twin and Fintech services for sustainable supply chain","award":["CUP B69J23000500005"],"award-info":[{"award-number":["CUP B69J23000500005"]}]},{"DOI":"10.13039\/501100007706","name":"Ministero dello Sviluppo Economico","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007706","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Autonomus unmmaneD vehicles in mArine environmental monitoring","award":["PE00000013_1 - CUP E63C22002150007"],"award-info":[{"award-number":["PE00000013_1 - CUP E63C22002150007"]}]},{"name":"Future Artificial Intelligence Research"},{"name":"National Recovery and Resilience Plan"},{"name":"Italian Ministry of University and Research"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,10,31]]},"abstract":"<jats:p>The increasing use of black-box networks in high-risk contexts has led researchers to propose explainable methods to make these networks transparent. Most methods that allow us to understand the behavior of Deep Neural Networks (DNNs) are post-hoc approaches, implying that the explainability is questionable, as these methods do not clarify the internal behavior of a model. Thus, this demonstrates the difficulty of interpreting the internal behavior of deep models. This systematic literature review collects the ante-hoc methods that provide an understanding of the internal mechanisms of deep models and which can be helpful to researchers who need to use interpretability methods to clarify DNNs. This work provides the definitions of strong interpretability and weak interpretability, which will be used to describe the interpretability of the methods discussed in this article. The results of this work are divided mainly into prototype-based methods, concept-based methods, and other interpretability methods for deep models.<\/jats:p>","DOI":"10.1145\/3728637","type":"journal-article","created":{"date-parts":[[2025,4,8]],"date-time":"2025-04-08T11:44:54Z","timestamp":1744112694000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Ante-Hoc Methods for Interpretable Deep Models: A Survey"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5748-2838","authenticated-orcid":false,"given":"Antonio","family":"Di Marino","sequence":"first","affiliation":[{"name":"Institute for High-Performance Computing and Networking (ICAR), Italian National Research Council (CNR), Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7139-3222","authenticated-orcid":false,"given":"Vincenzo","family":"Bevilacqua","sequence":"additional","affiliation":[{"name":"Institute for High-Performance Computing and Networking (ICAR), Italian National Research Council (CNR), Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5592-7995","authenticated-orcid":false,"given":"Angelo","family":"Ciaramella","sequence":"additional","affiliation":[{"name":"University of Naples Parthenope, Napoli, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6127-1195","authenticated-orcid":false,"given":"Ivanoe","family":"De Falco","sequence":"additional","affiliation":[{"name":"Institute for High-Performance Computing and Networking (ICAR), Italian National Research Council (CNR), Napoli, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7856-8761","authenticated-orcid":false,"given":"Giovanna","family":"Sannino","sequence":"additional","affiliation":[{"name":"Institute for High-Performance Computing and Networking (ICAR), Italian National Research Council (CNR), Napoli, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,5,7]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2022.3146552"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","unstructured":"Gulsum Alicioglu and Bo Sun. 2022. A survey of visual analytics for explainable artificial intelligence methods. Computers & Graphics 102 (Feb. 2022) 502\u2013520. 10.1016\/j.cag.2021.09.002","DOI":"10.1016\/j.cag.2021.09.002"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/3327757.3327875"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130140"},{"key":"e_1_3_2_6_2","unstructured":"Eslam Mohamed Bakr Liangbing Zhao Vincent Tao Hu Matthieu Cord Patrick Perez and Mohamed Elhoseiny. 2023. ToddlerDiffusion: Flash interpretable controllable diffusion model. arXiv:2311.14542. Retrieved from https:\/\/arxiv.org\/abs\/2311.14542"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i6.20551"},{"issue":"263","key":"e_1_3_2_8_2","first-page":"565","article-title":"A statistically precise and relatively simple method of estimating the bio-assay with quantal response, based on the logistic function","volume":"48","author":"Berkson Joseph","year":"1953","unstructured":"Joseph Berkson. 1953. A statistically precise and relatively simple method of estimating the bio-assay with quantal response, based on the logistic function. Journal of the American Statistical Association 48, 263 (1953), 565\u2013599.","journal-title":"Journal of the American Statistical Association"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01008"},{"key":"e_1_3_2_10_2","volume-title":"Classification and Regression Trees","author":"Breiman Leo","year":"1984","unstructured":"Leo Breiman (Ed.). 1984. Classification and Regression Trees. Wadsworth International Group, Belmont, CA, USA."},{"key":"e_1_3_2_11_2","volume-title":"Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (the Addison-Wesley Series in Artificial Intelligence)","author":"Buchanan Bruce G.","year":"1984","unstructured":"Bruce G. Buchanan and Edward H. Shortliffe. 1984. Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (the Addison-Wesley Series in Artificial Intelligence). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.3390\/make3040048"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788613"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/UIC-ATC.2017.8397411"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1177\/1473871620904671"},{"key":"e_1_3_2_16_2","unstructured":"Hila Chefer Oran Lang Mor Geva Volodymyr Polosukhin Assaf Shocher michal Irani Inbar Mosseri and Lior Wolf. 2024. The hidden language of diffusion models. In Event 12th International Conference on Learning Representations ICLR Hybrid Vienna Austria 20 pages."},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","author":"Chen Chaofan","year":"2019","unstructured":"Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. 2019. This looks like that: Deep learning for interpretable image recognition. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 801, 12 pages."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00265-z"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","unstructured":"Keyang Cheng Ning Wang and Maozhen Li. 2021. Interpretability of deep learning: A survey. In Advances in Natural Computation Fuzzy Systems and Knowledge Discovery Hongying Meng Tao Lei Maozhen Li Kenli Li Ning Xiong and Lipo Wang (Eds.). Springer International Publishing Cham 475\u2013486. 10.1007\/978-3-030-70665-4_54","DOI":"10.1007\/978-3-030-70665-4_54"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3645103"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50023-2"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1391"},{"key":"e_1_3_2_23_2","unstructured":"Miles Cranmer Alvaro Sanchez Gonzalez Peter Battaglia Rui Xu Kyle Cranmer David Spergel and Shirley Ho. 2020. Discovering symbolic models from deep learning with inductive biases. In Advances in Neural Information Processing Systems H. Larochelle M. Ranzato R. Hadsell M. F. Balcan and H. Lin (Eds.). Vol. 33. Curran Associates Inc. 17429\u201317442. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/hash\/c9f2f917078bd2db12f23c3b413d9cba-Abstract.html"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-47954-6"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2022.10.013"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.23919\/MIPRO.2018.8400040"},{"key":"e_1_3_2_28_2","volume-title":"Regulation (EU) 2016\/679 of the European Parliament and of the Council","author":"Parliament European","year":"2016","unstructured":"European Parliament and Council of the European Union. 2016. Regulation (EU) 2016\/679 of the European Parliament and of the Council. Official Journal of the European Union. Retrieved February 15, 2024 from https:\/\/eur-lex.europa.eu\/eli\/reg\/2016\/679\/2016-05-04"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRPMS.2021.3066428"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206772"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2024.3459626"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v38i3.2741"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1002\/ail2.61"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"Linde S. Hesse Nicola K. Dinsdale and Ana I. L. Namburete. 2024. Prototype learning for explainable brain age prediction. In Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE Waikoloa Hawaii 7903\u20137913.","DOI":"10.1109\/WACV57701.2024.00772"},{"key":"e_1_3_2_37_2","first-page":"6840","article-title":"Denoising diffusion probabilistic models","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Proceedings of the 34th International Conference on Neural Information Processing Systems. 6840\u20136851.","journal-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","unstructured":"Robert C. Holte. 1993. Very simple classification rules perform well on most commonly used datasets. Machine Learning 11 1 (1993) 63\u201390. 10.1023\/A:1022631118932","DOI":"10.1023\/A:1022631118932"},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","unstructured":"Jinyung Hong Keun Hee Park and Theodore P. Pavlic. 2024. Concept-centric transformers: Enhancing model interpretability through object-centric concept learning within a shared global workspace. In Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE Waikoloa Hawaii 4880\u20134891.","DOI":"10.1109\/WACV57701.2024.00481"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i19.30109"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","unstructured":"Maksims Ivanovs Roberts Kadikis and Kaspars Ozols. 2021. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters 150 (Oct. 2021) 228\u2013234. 10.1016\/j.patrec.2021.06.030","DOI":"10.1016\/j.patrec.2021.06.030"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1002\/wsbm.1548"},{"key":"e_1_3_2_44_2","unstructured":"Dmitry Kazhdan Botty Dimanov Mateja Jamnik Pietro Li\u00f2 and Adrian Weller. 2020. Now you see me (CME): Concept-based model extraction. arxiv:2010.13233. Retrieved from https:\/\/arxiv.org\/abs\/2010.13233"},{"key":"e_1_3_2_45_2","series-title":"Proceedings of Machine Learning Research","first-page":"11162","volume-title":"Proceedings of the 39th International Conference on Machine Learning","volume":"162","author":"Kim Sangwon","year":"2022","unstructured":"Sangwon Kim, Jaeyeal Nam, and Byoung Chul Ko. 2022. ViT-NeT: Interpretable vision transformers with neural tree decoder. In Proceedings of the 39th International Conference on Machine Learning. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.), Proceedings of Machine Learning Research, Vol. 162, PMLR, Baltimore, Maryland, USA, 11162\u201311172. Retrieved from https:\/\/proceedings.mlr.press\/v162\/kim22g.html"},{"issue":"2004","key":"e_1_3_2_46_2","first-page":"1","article-title":"Procedures for performing systematic reviews","volume":"33","author":"Kitchenham Barbara","year":"2004","unstructured":"Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1\u201326.","journal-title":"Keele, UK, Keele University"},{"key":"e_1_3_2_47_2","series-title":"Proceedings of Machine Learning Research","first-page":"5338","volume-title":"Proceedings of the 37th International Conference on Machine Learning (ICML\u201920)","volume":"119","author":"Koh Pang Wei","year":"2020","unstructured":"Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning (ICML\u201920)(Proceedings of Machine Learning Research, Vol. 119). PMLR, Virtual Event, 5338\u20135348."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","unstructured":"Boris Kovalerchuk Muhammad Aurangzeb Ahmad and Ankur Teredesai. 2021. Survey of explainable machine learning with visual and granular methods beyond quasi-explanations. Studies in Computational Intelligence 937 (Mar 2021) 217\u2013267. 10.1007\/978-3-030-64949-4_8","DOI":"10.1007\/978-3-030-64949-4_8"},{"key":"e_1_3_2_49_2","volume-title":"Can A.I. Be Taught to Explain Itself?","author":"Kuang Cliff","year":"2017","unstructured":"Cliff Kuang. 2017. Can A.I. Be Taught to Explain Itself? The New York Times. Retrieved February 15, 2024 from https:\/\/www.nytimes.com\/2017\/11\/21\/magazine\/can-ai-be-taught-to-explain-itself.html"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.140"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOAS848"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00108"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11771"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-022-01756-8"},{"issue":"1","key":"e_1_3_2_56_2","first-page":"29","article-title":"A survey of data-driven and knowledge-aware explainable AI","volume":"34","author":"Li Xiao-Hui","year":"2020","unstructured":"Xiao-Hui Li, Caleb Chen Cao, Yuhan Shi, Wei Bai, Han Gao, Luyu Qiu, Cong Wang, Yuanyuan Gao, Shenjia Zhang, Xun Xue, et\u00a0al. 2020. A survey of data-driven and knowledge-aware explainable AI. IEEE Transactions on Knowledge and Data Engineering 34, 1 (2020), 29\u201349.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","unstructured":"Yu Liang Siguang Li Chungang Yan Maozhen Li and Changjun Jiang. 2021. Explaining the black-box model: A survey of local interpretation methods for deep neural networks. Neurocomputing 419 (Jan. 2021) 168\u2013182. 10.1016\/j.neucom.2020.08.011","DOI":"10.1016\/j.neucom.2020.08.011"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.3390\/e23010018"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","unstructured":"Peter Lipton. 1990. Contrastive explanation. Royal Institute of Philosophy Supplement 27 (March 1990) 247\u2013266. 10.1017\/S1358246100005130","DOI":"10.1017\/S1358246100005130"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_2_61_2","first-page":"11525","article-title":"Object-centric learning with slot attention","author":"Locatello Francesco","year":"2020","unstructured":"Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot attention. In Proceedings of the 34th International Conference on Neural Information Processing Systems. 11525\u201311538.","journal-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339556"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295230"},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","unstructured":"Jian-Xun Mi An-Di Li and Li-Fang Zhou. 2020. Review study of interpretation methods for future interpretable machine learning. IEEE Access 8 (Oct. 2020) 191969\u2013191985.","DOI":"10.1109\/ACCESS.2020.3032756"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","unstructured":"Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (Feb. 2019) 1\u201338. 10.1016\/j.artint.2018.07.007","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_3_2_66_2","volume-title":"Interpretable Machine Learning","author":"Molnar Christoph","year":"2019","unstructured":"Christoph Molnar. 2019. Interpretable Machine Learning. Lulu.com, Research Triangle, NC, USA."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","unstructured":"Meike Nauta Annemarie Jutte Jesper Provoost and Christin Seifert. 2021. This looks like that because ... explaining prototypes for interpretable image recognition. In Machine Learning and Principles and Practice of Knowledge Discovery in Databases Michael Kamp Irena Koprinska and Adrien Bibal et\u00a0al. (Eds.). Springer International Publishing Cham 441\u2013456. 10.1007\/978-3-030-93736-2_34","DOI":"10.1007\/978-3-030-93736-2_34"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.106668"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-63787-2_2"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3674837"},{"key":"e_1_3_2_71_2","unstructured":"Vitali Petsiuk Abir Das and Kate Saenko. 2018. RISE: Randomized input sampling for explanation of black-box models. arXiv:1806.07421. Retrieved from https:\/\/arxiv.org\/abs\/1806.07421"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","unstructured":"R. Quinlan. 1986. Induction of decision trees. Machine Learning 1 1 (March 1986) 81\u2013106. 10.1007\/BF00116251","DOI":"10.1007\/BF00116251"},{"key":"e_1_3_2_73_2","first-page":"8748","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning. PMLR, Virtual Event, 8748\u20138763. Retrieved from https:\/\/proceedings.mlr.press\/v139\/radford21a.html"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00075"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_76_2","first-page":"16","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR\u201922)","author":"Rigotti Mattia","year":"2022","unstructured":"Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. 2022. Attention-based interpretability with concept transformers. In Proceedings of the International Conference on Learning Representations (ICLR\u201922). OpenReview.net, Virtual Event, 16 pages."},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0048-x"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","unstructured":"Cynthia Rudin Chaofan Chen Zhi Chen Haiyang Huang Lesia Semenova and Chudi Zhong. 2022. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys 16 none (Jan. 2022). 10.1214\/21-SS133","DOI":"10.1214\/21-SS133"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","unstructured":"Waddah Saeed and Christian Omlin. 2023. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems 263 (March 2023) 110273. 10.1016\/j.knosys.2023.110273","DOI":"10.1016\/j.knosys.2023.110273"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01004"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01228-7"},{"key":"e_1_3_2_83_2","first-page":"76737","article-title":"Interpretable prototype-based graph information bottleneck","author":"Seo Sangwoo","year":"2023","unstructured":"Sangwoo Seo, Sungwon Kim, and Chanyoung Park. 2023. Interpretable prototype-based graph information bottleneck. In Proceedings of the 37th International Conference on Neural Information Processing Systems. 76737\u201376748. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/hash\/f224f056694bcfe465c5d84579785761-Abstract-Conference.html","journal-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1282"},{"key":"e_1_3_2_85_2","series-title":"ICML\u201917","first-page":"3145","volume-title":"Proceedings of the 34th International Conference on Machine Learning","volume":"70","author":"Shrikumar Avanti","year":"2017","unstructured":"Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning(ICML\u201917, Vol. 70). JMLR.org, 3145\u20133153. Retrieved from https:\/\/proceedings.mlr.press\/v70\/shrikumar17a.html"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"e_1_3_2_87_2","unstructured":"Karen Simonyan Andrea Vedaldi and Andrew Zisserman. 2014. Deep inside convolutional networks: visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations ICLR Banff AB Canada 8 pages."},{"key":"e_1_3_2_88_2","doi-asserted-by":"crossref","unstructured":"Gurmail Singh and Kin-Choong Yow. 2021. These do not look like those: An interpretable deep learning model for image recognition. IEEE Access 9 (March 2021) 41482\u201341493.","DOI":"10.1109\/ACCESS.2021.3064838"},{"key":"e_1_3_2_89_2","doi-asserted-by":"crossref","unstructured":"Mohana Singh B. S. Vivek Jayavardhana Gubbi and Arpan Pal. 2024. Prototype-based interpretable model for glaucoma detection. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE Seattle WA USA 5056\u20135065.","DOI":"10.1109\/CVPRW63382.2024.00512"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-73016-0_8"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-97-2259-4_5"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00530-022-00960-4"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.3027314"},{"issue":"4","key":"e_1_3_2_96_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3705724","article-title":"Explainable artificial intelligence: Importance, use domains, stages, output shapes, and challenges","volume":"57","author":"Ullah Naeem","year":"2024","unstructured":"Naeem Ullah, Javed Ali Khan, Ivanoe De Falco, and Giovanna Sannino. 2024. Explainable artificial intelligence: Importance, use domains, stages, output shapes, and challenges. ACM Computing Surveys 57, 4 (2024), 1\u201336.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","unstructured":"Bas H. M. Van Der Velden Hugo J. Kuijf Kenneth G. A. Gilhuijs and Max A. Viergever. 2022. Explainable artificial intelligence (XAI) in deep learning-based medical image analysis. Medical Image Analysis 79 (July 2022) 102470. 10.1016\/j.media.2022.102470","DOI":"10.1016\/j.media.2022.102470"},{"key":"e_1_3_2_98_2","article-title":"Attention is all you need","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_99_2","volume-title":"The Caltech-UCSD Birds-200-2011 Dataset","author":"Wah C.","year":"2011","unstructured":"C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001. California Institute of Technology."},{"key":"e_1_3_2_100_2","unstructured":"Qiyang Wan Ruiping Wang and Xilin Chen. 2024. Interpretable object recognition by semantic prototype analysis. In Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE Waikoloa Hawaii 800\u2013809."},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN54540.2023.10191863"},{"key":"e_1_3_2_102_2","volume-title":"Full Translation: China\u2019s \u2018New Generation Artificial Intelligence Development Plan\u2019","author":"Webster Graham","year":"2017","unstructured":"Graham Webster, Rogier Creemers, Elsa Kania, and Paul Triolo. 2017. Full Translation: China\u2019s \u2018New Generation Artificial Intelligence Development Plan\u2019. DigiChina. Retrieved February 15, 2024 from https:\/\/digichina.stanford.edu\/work\/full-translation-chinas-new-generation-artificial-intelligence-development-plan-2017\/"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-32236-6_51"},{"key":"e_1_3_2_104_2","unstructured":"Xinyue Xu Yi Qin Lu Mi Hao Wang and Xiaomeng Li. 2024. Energy-Based concept bottleneck models: unifying prediction concept intervention and probabilistic interpretations. In International Conference on Learning Representations ICLR. Vienna Austria 23 pages."},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1049\/ipr2.13074"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i9.26267"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-022-01657-x"},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700808"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2021.3100641"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","unstructured":"Zhibo Zhang Hussam Al Hamadi Ernesto Damiani Chan Yeob Yeun and Fatma Taher. 2022. Explainable artificial intelligence applications in cyber security: State-of-the-art in research. IEEE Access 10 (Sept. 2022) 93104\u201393139. 10.1109\/ACCESS.2022.3204051","DOI":"10.1109\/ACCESS.2022.3204051"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i8.20898"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.319"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1145\/3529755"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728637","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728637","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:36Z","timestamp":1750295916000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728637"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,7]]},"references-count":112,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2025,10,31]]}},"alternative-id":["10.1145\/3728637"],"URL":"https:\/\/doi.org\/10.1145\/3728637","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,7]]},"assertion":[{"value":"2024-06-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}