{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T10:39:37Z","timestamp":1792751977659,"version":"4.5.0"},"reference-count":142,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2018,8,22]],"date-time":"2018-08-22T00:00:00Z","timestamp":1534896000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2019,9,30]]},"abstract":"<jats:p>In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.<\/jats:p>","DOI":"10.1145\/3236009","type":"journal-article","created":{"date-parts":[[2018,8,22]],"date-time":"2018-08-22T08:41:46Z","timestamp":1534927306000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4131,"title":["A Survey of Methods for Explaining Black Box Models"],"prefix":"10.1145","volume":"51","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2827-7613","authenticated-orcid":false,"given":"Riccardo","family":"Guidotti","sequence":"first","affiliation":[{"name":"KDDLab, University of Pisa, Pisa, PI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna","family":"Monreale","sequence":"additional","affiliation":[{"name":"KDDLab, University of Pisa, Pisa, PI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salvatore","family":"Ruggieri","sequence":"additional","affiliation":[{"name":"KDDLab, University of Pisa, Pisa, PI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Franco","family":"Turini","sequence":"additional","affiliation":[{"name":"KDDLab, University of Pisa, Pisa, PI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fosca","family":"Giannotti","sequence":"additional","affiliation":[{"name":"KDDLab, ISTI-CNR, Pisa, PI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dino","family":"Pedreschi","sequence":"additional","affiliation":[{"name":"KDDLab, University of Pisa, Pisa, PI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,8,22]]},"reference":[{"key":"e_1_2_3_1_1","unstructured":"Julius Adebayo and Lalana Kagal. 2016. Iterative orthogonal feature projection for diagnosing bias in black-box models. arXiv preprint arXiv:1611.04967."},{"key":"e_1_2_3_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0011"},{"key":"e_1_2_3_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/645920.672836"},{"key":"e_1_2_3_4_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40064-015-1481-x"},{"key":"e_1_2_3_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/0950-7051(96)81920-4"},{"key":"e_1_2_3_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-011-9207-8"},{"key":"e_1_2_3_7_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130140"},{"key":"e_1_2_3_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859912"},{"key":"e_1_2_3_9_1","doi-asserted-by":"publisher","DOI":"10.1214\/11-AOAS495"},{"key":"e_1_2_3_10_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022685808937"},{"key":"e_1_2_3_11_1","volume-title":"VisualBackProp: Visualizing CNNs for autonomous driving. CoRR","author":"Bojarski Mariusz","year":"2016","unstructured":"Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba. 2016. VisualBackProp: Visualizing CNNs for autonomous driving. CoRR, Vol. abs\/1611.05418 (2016)."},{"key":"e_1_2_3_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775113"},{"key":"e_1_2_3_13_1","volume-title":"Olshen","author":"Breiman Leo","year":"1984","unstructured":"Leo Breiman, Jerome Friedman, Charles J. Stone, and Richard A. Olshen. 1984. Classification and Regression Trees. CRC Press."},{"key":"e_1_2_3_14_1","volume-title":"Semantics derived automatically from language corpora necessarily contain human biases. arXiv preprint arXiv:1608.07187","author":"Caliskan-Islam Aylin","year":"2016","unstructured":"Aylin Caliskan-Islam, Joanna J. Bryson, and Arvind Narayanan. 2016. Semantics derived automatically from language corpora necessarily contain human biases. arXiv preprint arXiv:1608.07187 (2016)."},{"key":"e_1_2_3_15_1","first-page":"23","article-title":"The credit card market and regulation: In need of repair","volume":"10","author":"Carter Carolyn","year":"2006","unstructured":"Carolyn Carter, Elizabeth Renuart, Margot Saunders, and Chi Chi Wu. 2006. The credit card market and regulation: In need of repair. NC Bank. Inst. 10 (2006), 23.","journal-title":"NC Bank. Inst."},{"key":"e_1_2_3_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788613"},{"key":"e_1_2_3_17_1","volume-title":"Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92","author":"Chipman H. A.","unstructured":"H. A. Chipman, E. I. George, and R. E. McCulloh. 1998. Making sense of a forest of trees. In Proceedings of the 30th Symposium on the Interface, S. Weisberg (Ed.). Fairfax Station, VA: Interface Foundation of North America, 84--92."},{"key":"e_1_2_3_18_1","volume-title":"Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM\u201911)","author":"Cortez Paulo","unstructured":"Paulo Cortez and Mark J. Embrechts. 2011. Opening black box data mining models using sensitivity analysis. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM\u201911). IEEE, 341--348."},{"key":"e_1_2_3_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2012.10.039"},{"key":"e_1_2_3_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04747-3_8"},{"key":"e_1_2_3_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/3091574.3091580"},{"key":"e_1_2_3_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2998828.2998832"},{"key":"e_1_2_3_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2017.1"},{"key":"e_1_2_3_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.42"},{"key":"e_1_2_3_25_1","volume-title":"Interpreting tree ensembles with intrees. arXiv preprint arXiv:1408.5456","author":"Deng Houtao","year":"2014","unstructured":"Houtao Deng. 2014. Interpreting tree ensembles with intrees. arXiv preprint arXiv:1408.5456 (2014)."},{"key":"e_1_2_3_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/2639331.2639334"},{"key":"e_1_2_3_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/3000292.3000299"},{"key":"e_1_2_3_28_1","unstructured":"Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv:1702.08608v2."},{"key":"e_1_2_3_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1756007"},{"key":"e_1_2_3_30_1","volume-title":"Interpretable explanations of black boxes by meaningful perturbation. arXiv preprint arXiv:1704.03296","author":"Fong Ruth","year":"2017","unstructured":"Ruth Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturbation. arXiv preprint arXiv:1704.03296 (2017)."},{"key":"e_1_2_3_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/645527.657305"},{"key":"e_1_2_3_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594473.2594475"},{"key":"e_1_2_3_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081878"},{"key":"e_1_2_3_34_1","doi-asserted-by":"publisher","DOI":"10.4088\/JCP.12m08338"},{"key":"e_1_2_3_35_1","doi-asserted-by":"publisher","DOI":"10.1080\/10618600.2014.907095"},{"key":"e_1_2_3_36_1","volume-title":"Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI\u201916)","author":"Goodman Bryce","year":"2016","unstructured":"Bryce Goodman and Seth Flaxman. 2016. EU regulations on algorithmic decision-making and a \u201cright to explanation.\u201d In Proceedings of the ICML Workshop on Human Interpretability in Machine Learning (WHI\u201916). Retrieved from http:\/\/arxiv. org\/abs\/1606.08813 v1."},{"key":"e_1_2_3_37_1","volume-title":"Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820","author":"Guidotti Riccardo","year":"2018","unstructured":"Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. 2018. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018)."},{"key":"e_1_2_3_38_1","volume-title":"Making tree ensembles interpretable. arXiv preprint arXiv:1606.05390","author":"Hara Satoshi","year":"2016","unstructured":"Satoshi Hara and Kohei Hayashi. 2016. Making tree ensembles interpretable. arXiv preprint arXiv:1606.05390 (2016)."},{"key":"e_1_2_3_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2013.10.067"},{"key":"e_1_2_3_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-014-0368-8"},{"key":"e_1_2_3_41_1","volume-title":"Watts","author":"Hofman Jake M.","year":"2017","unstructured":"Jake M. Hofman, Amit Sharma, and Duncan J. Watts. 2017. Prediction and explanation in social systems. Science 355, 6324 (2017), 486--488."},{"key":"e_1_2_3_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014122"},{"key":"e_1_2_3_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2010.12.003"},{"key":"e_1_2_3_44_1","volume-title":"Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16","author":"Johansson U.","unstructured":"U. Johansson, R. K\u00f6nig, and L. Niklasson. 2003. Rule extraction from trained neural networks using genetic programming. In Proceedings of the 13th International Conference on Artificial Neural Networks. 13--16."},{"key":"e_1_2_3_45_1","volume-title":"Proceedings of the FLAIRS Conference. 658--663","author":"Johansson Ulf","year":"2004","unstructured":"Ulf Johansson, Rikard K\u00f6nig, and Lars Niklasson. 2004. The truth is in there-rule extraction from opaque models using genetic programming. In Proceedings of the FLAIRS Conference. 658--663."},{"key":"e_1_2_3_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CIDM.2009.4938655"},{"key":"e_1_2_3_47_1","volume-title":"Proceedings of the 7th International Conference on Information Fusion","volume":"1","author":"Johansson Ulf","year":"2004","unstructured":"Ulf Johansson, Lars Niklasson, and Rikard K\u00f6nig. 2004. Accuracy vs. comprehensibility in data mining models. In Proceedings of the 7th International Conference on Information Fusion, Vol. 1. 295--300."},{"key":"e_1_2_3_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.127"},{"key":"e_1_2_3_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/3157096.3157352"},{"key":"e_1_2_3_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969033.2969045"},{"key":"e_1_2_3_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969442.2969492"},{"key":"e_1_2_3_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-47175-4_20"},{"key":"e_1_2_3_54_1","volume-title":"Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730","author":"Koh Pang Wei","year":"2017","unstructured":"Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730 (2017)."},{"key":"e_1_2_3_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_2_3_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2016.2628365"},{"key":"e_1_2_3_57_1","doi-asserted-by":"crossref","unstructured":"R. Krishnan G. Sivakumar and P. Bhattacharya. 1999. Extracting decision trees from trained neural networks. Pattern Recogn. 32 12 (1999).","DOI":"10.1016\/S0031-3203(98)00181-2"},{"key":"e_1_2_3_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077257.3077271"},{"key":"e_1_2_3_59_1","first-page":"633","article-title":"Accountable algorithms","volume":"165","author":"Kroll Joshua A.","year":"2017","unstructured":"Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. U. Penn. Law Rev. 165 (2017), 633--705.","journal-title":"U. Penn. Law Rev."},{"key":"e_1_2_3_60_1","volume-title":"Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533","author":"Kurakin Alexey","year":"2016","unstructured":"Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 (2016)."},{"key":"e_1_2_3_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939874"},{"key":"e_1_2_3_62_1","volume-title":"Interpretable 8 explorable approximations of black box models. arXiv preprint arXiv:1707.01154","author":"Lakkaraju Himabindu","year":"2017","unstructured":"Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2017. Interpretable 8 explorable approximations of black box models. arXiv preprint arXiv:1707.01154 (2017)."},{"key":"e_1_2_3_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098066"},{"key":"e_1_2_3_64_1","volume-title":"Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM\u201913)","author":"Landecker Will","unstructured":"Will Landecker, Michael D. Thomure, Lu\u00eds M. A. Bettencourt, Melanie Mitchell, Garrett T. Kenyon, and Steven P. Brumby. 2013. Interpreting individual classifications of hierarchical networks. In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM\u201913). IEEE, 32--38."},{"key":"e_1_2_3_65_1","volume-title":"Rationalizing neural predictions. arXiv preprint arXiv:1606.04155","author":"Lei Tao","year":"2016","unstructured":"Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. arXiv preprint arXiv:1606.04155 (2016)."},{"key":"e_1_2_3_66_1","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOAS848"},{"key":"e_1_2_3_67_1","volume-title":"Deep text classification can be fooled. arXiv preprint arXiv:1704.08006","author":"Liang Bin","year":"2017","unstructured":"Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2017. Deep text classification can be fooled. arXiv preprint arXiv:1704.08006 (2017)."},{"key":"e_1_2_3_68_1","volume-title":"The mythos of model interpretability. arXiv preprint arXiv:1606.03490","author":"Lipton Zachary C.","year":"2016","unstructured":"Zachary C. Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016)."},{"key":"e_1_2_3_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339556"},{"key":"e_1_2_3_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487579"},{"key":"e_1_2_3_71_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.296.6623.657"},{"key":"e_1_2_3_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299155"},{"key":"e_1_2_3_73_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0911-8"},{"key":"e_1_2_3_74_1","doi-asserted-by":"publisher","DOI":"10.1093\/idpl\/ipx019"},{"key":"e_1_2_3_75_1","volume-title":"Transparent Data Mining for Big and Small Data","author":"Malioutov Dmitry M.","unstructured":"Dmitry M. Malioutov, Kush R. Varshney, Amin Emad, and Sanjeeb Dash. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer, 95--121."},{"key":"e_1_2_3_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2006.04.051"},{"key":"e_1_2_3_77_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2011.01.013"},{"key":"e_1_2_3_78_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2016.11.008"},{"key":"e_1_2_3_79_1","doi-asserted-by":"publisher","DOI":"10.5555\/3091825.3091863"},{"key":"e_1_2_3_80_1","doi-asserted-by":"publisher","DOI":"10.5555\/3157382.3157477"},{"key":"e_1_2_3_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"e_1_2_3_82_1","volume-title":"Proceedings of the European Symposium on Artificial Neural Networks (ESANN\u201902)","author":"N\u00fa\u00f1ez Haydemar","year":"2002","unstructured":"Haydemar N\u00fa\u00f1ez, Cecilio Angulo, and Andreu Catal\u00e0. 2002. Rule extraction from support vector machines. In Proceedings of the European Symposium on Artificial Neural Networks (ESANN\u201902). 107--112."},{"key":"e_1_2_3_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0304-3800(02)00064-9"},{"key":"e_1_2_3_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463372.2463382"},{"key":"e_1_2_3_85_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bti1102"},{"key":"e_1_2_3_86_1","doi-asserted-by":"publisher","DOI":"10.5555\/2717112"},{"key":"e_1_2_3_87_1","first-page":"380","article-title":"Acceptance of rules generated by machine learning among medical experts. Methods Info","volume":"40","author":"Pazzani Michael J.","year":"2001","unstructured":"Michael J. Pazzani, S. Mani, William R. Shankle et al. 2001. Acceptance of rules generated by machine learning among medical experts. Methods Info. Med. 40, 5 (2001), 380--385.","journal-title":"Med."},{"key":"e_1_2_3_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401959"},{"key":"e_1_2_3_89_1","doi-asserted-by":"publisher","DOI":"10.5555\/1597122.1597143"},{"key":"e_1_2_3_90_1","doi-asserted-by":"publisher","DOI":"10.5555\/1625015.1625078"},{"key":"e_1_2_3_91_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-7373(87)80053-6"},{"key":"e_1_2_3_92_1","doi-asserted-by":"publisher","DOI":"10.5555\/152181"},{"key":"e_1_2_3_93_1","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1987.0321"},{"key":"e_1_2_3_94_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-56602-3_124"},{"key":"e_1_2_3_95_1","volume-title":"Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444","author":"Radford Alec","year":"2017","unstructured":"Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. 2017. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444 (2017)."},{"key":"e_1_2_3_96_1","volume-title":"Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386","author":"Ribeiro Marco Tulio","year":"2016","unstructured":"Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016)."},{"key":"e_1_2_3_97_1","volume-title":"Nothing else matters: Model-agnostic explanations by identifying prediction invariance. arXiv preprint arXiv:1611.05817","author":"Ribeiro Marco Tulio","year":"2016","unstructured":"Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Nothing else matters: Model-agnostic explanations by identifying prediction invariance. arXiv preprint arXiv:1611.05817 (2016)."},{"key":"e_1_2_3_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_3_99_1","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI\u201918)","author":"Ribeiro Marco Tulio","year":"2018","unstructured":"Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI\u201918)."},{"key":"e_1_2_3_100_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888913000039"},{"key":"e_1_2_3_101_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972825.33"},{"key":"e_1_2_3_102_1","doi-asserted-by":"publisher","DOI":"10.1111\/0272-4332.00040"},{"key":"e_1_2_3_103_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2016.2599820"},{"key":"e_1_2_3_104_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITB.2006.880553"},{"key":"e_1_2_3_105_1","volume-title":"Transparent Data Mining for Big and Small Data","author":"Seifert Christin","unstructured":"Christin Seifert, Aisha Aamir, Aparna Balagopalan, Dhruv Jain, Abhinav Sharma, Sebastian Grottel, and Stefan Gumhold. 2017. Visualizations of deep neural networks in computer vision: A survey. In Transparent Data Mining for Big and Small Data. Springer, 123--144."},{"key":"e_1_2_3_106_1","volume-title":"Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. arXiv preprint arXiv:1610.02391","author":"Selvaraju Ramprasaath R.","year":"2016","unstructured":"Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. 2016. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. arXiv preprint arXiv:1610.02391 (2016)."},{"key":"e_1_2_3_107_1","volume-title":"Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685","author":"Shrikumar Avanti","year":"2017","unstructured":"Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685 (2017)."},{"key":"e_1_2_3_108_1","volume-title":"Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810","author":"Shwartz-Ziv Ravid","year":"2017","unstructured":"Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)."},{"key":"e_1_2_3_109_1","volume-title":"Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034","author":"Simonyan Karen","year":"2013","unstructured":"Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)."},{"key":"e_1_2_3_110_1","volume-title":"Marco Tulio Ribeiro, and Carlos Guestrin","author":"Singh Sameer","year":"2016","unstructured":"Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin. 2016. Programs as black-box explanations. arXiv preprint arXiv:1611.07579 (2016)."},{"key":"e_1_2_3_111_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btn170"},{"key":"e_1_2_3_112_1","volume-title":"Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806","author":"Springenberg Jost Tobias","year":"2014","unstructured":"Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. 2014. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014)."},{"key":"e_1_2_3_113_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jneumeth.2016.10.008"},{"key":"e_1_2_3_114_1","volume-title":"Malioutov","author":"Su Guolong","year":"2015","unstructured":"Guolong Su, Dennis Wei, Kush R. Varshney, and Dmitry M. Malioutov. 2015. Interpretable two-level Boolean rule learning for classification. arXiv preprint arXiv:1511.07361 (2015)."},{"key":"e_1_2_3_115_1","volume-title":"Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365","author":"Sundararajan Mukund","year":"2017","unstructured":"Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365 (2017)."},{"key":"e_1_2_3_116_1","volume-title":"Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199","author":"Szegedy Christian","year":"2013","unstructured":"Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)."},{"key":"e_1_2_3_117_1","volume-title":"Wells","author":"Tan Hui Fen","year":"2016","unstructured":"Hui Fen Tan, Giles Hooker, and Martin T. Wells. 2016. Tree space prototypes: Another look at making tree ensembles interpretable. arXiv preprint arXiv:1611.07115 (2016)."},{"key":"e_1_2_3_118_1","unstructured":"Pang-Ning Tan et al. 2006. Introduction to Data Mining. Pearson Education India."},{"key":"e_1_2_3_119_1","volume-title":"TreeView: Peeking into deep neural networks via feature-space partitioning. arXiv preprint arXiv:1611.07429","author":"Thiagarajan Jayaraman J.","year":"2016","unstructured":"Jayaraman J. Thiagarajan, Bhavya Kailkhura, Prasanna Sattigeri, and Karthikeyan Natesan Ramamurthy. 2016. TreeView: Peeking into deep neural networks via feature-space partitioning. arXiv preprint arXiv:1611.07429 (2016)."},{"key":"e_1_2_3_120_1","volume-title":"Explaining recommendations: Design and evaluation","author":"Tintarev Nava","unstructured":"Nava Tintarev and Judith Masthoff. 2015. Explaining recommendations: Design and evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 353--382."},{"key":"e_1_2_3_121_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098039"},{"key":"e_1_2_3_122_1","doi-asserted-by":"publisher","DOI":"10.1109\/MLSP.2016.7738872"},{"key":"e_1_2_3_123_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2010.08.023"},{"key":"e_1_2_3_124_1","volume-title":"Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567","author":"Vidovic Marina M.-C.","year":"2016","unstructured":"Marina M.-C. Vidovic, Nico G\u00f6rnitz, Klaus-Robert M\u00fcller, and Marius Kloft. 2016. Feature importance measure for non-linear learning algorithms. arXiv preprint arXiv:1611.07567 (2016)."},{"key":"e_1_2_3_125_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.8"},{"key":"e_1_2_3_126_1","doi-asserted-by":"publisher","DOI":"10.1093\/idpl\/ipx005"},{"key":"e_1_2_3_127_1","volume-title":"Proceedings of the Conference on Artificial Intelligence and Statistics. 1013--1022","author":"Wang Fulton","year":"2015","unstructured":"Fulton Wang and Cynthia Rudin. 2015. Falling rule lists. In Proceedings of the Conference on Artificial Intelligence and Statistics. 1013--1022."},{"key":"e_1_2_3_128_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783407"},{"key":"e_1_2_3_129_1","volume-title":"Multi-value rule sets. arXiv preprint arXiv:1710.05257","author":"Wang Tong","year":"2017","unstructured":"Tong Wang. 2017. Multi-value rule sets. arXiv preprint arXiv:1710.05257 (2017)."},{"key":"e_1_2_3_130_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0171"},{"key":"e_1_2_3_131_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995616"},{"key":"e_1_2_3_132_1","volume-title":"Challenges for transparency. arXiv preprint arXiv:1708.01870","author":"Weller Adrian","year":"2017","unstructured":"Adrian Weller. 2017. Challenges for transparency. arXiv preprint arXiv:1708.01870 (2017)."},{"key":"e_1_2_3_133_1","doi-asserted-by":"publisher","DOI":"10.5555\/273530.273548"},{"key":"e_1_2_3_134_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045336"},{"key":"e_1_2_3_135_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972733.40"},{"key":"e_1_2_3_136_1","volume-title":"Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579","author":"Yosinski Jason","year":"2015","unstructured":"Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)."},{"key":"e_1_2_3_137_1","volume-title":"Zeiler and Rob Fergus","author":"Matthew","year":"2014","unstructured":"Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European Conference on Computer Vision. Springer, 818--833."},{"key":"e_1_2_3_138_1","volume-title":"Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530","author":"Zhang Chiyuan","year":"2016","unstructured":"Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 (2016)."},{"key":"e_1_2_3_139_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.319"},{"key":"e_1_2_3_140_1","volume-title":"Interpreting models via single tree approximation. arXiv preprint arXiv:1610.09036","author":"Zhou Yichen","year":"2016","unstructured":"Yichen Zhou and Giles Hooker. 2016. Interpreting models via single tree approximation. arXiv preprint arXiv:1610.09036 (2016)."},{"key":"e_1_2_3_141_1","doi-asserted-by":"publisher","DOI":"10.5555\/859130.859132"},{"key":"e_1_2_3_142_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04174-7_45"},{"key":"e_1_2_3_143_1","volume-title":"Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595","author":"Zintgraf Luisa M.","year":"2017","unstructured":"Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling. 2017. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595 (2017)."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236009","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236009","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:57:25Z","timestamp":1750193845000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236009"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,8,22]]},"references-count":142,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,9,30]]}},"alternative-id":["10.1145\/3236009"],"URL":"https:\/\/doi.org\/10.1145\/3236009","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,8,22]]},"assertion":[{"value":"2018-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-08-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}