{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T16:16:21Z","timestamp":1781799381700,"version":"3.54.5"},"reference-count":120,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,12,8]],"date-time":"2021-12-08T00:00:00Z","timestamp":1638921600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Deep Learning is a state-of-the-art technique to make inference on extensive or complex data. As a black box model due to their multilayer nonlinear structure, Deep Neural Networks are often criticized as being non-transparent and their predictions not traceable by humans. Furthermore, the models learn from artificially generated datasets, which often do not reflect reality. By basing decision-making algorithms on Deep Neural Networks, prejudice and unfairness may be promoted unknowingly due to a lack of transparency. Hence, several so-called explanators, or explainers, have been developed. Explainers try to give insight into the inner structure of machine learning black boxes by analyzing the connection between the input and output. In this survey, we present the mechanisms and properties of explaining systems for Deep Neural Networks for Computer Vision tasks. We give a comprehensive overview about the taxonomy of related studies and compare several survey papers that deal with explainability in general. We work out the drawbacks and gaps and summarize further research ideas.<\/jats:p>","DOI":"10.3390\/make3040048","type":"journal-article","created":{"date-parts":[[2021,12,10]],"date-time":"2021-12-10T02:07:18Z","timestamp":1639102038000},"page":"966-989","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":313,"title":["Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey"],"prefix":"10.3390","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2356-1313","authenticated-orcid":false,"given":"Vanessa","family":"Buhrmester","sequence":"first","affiliation":[{"name":"Fraunhofer IOSB, Gutleuthausstra\u00dfe 1, 76275 Ettlingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8577-5256","authenticated-orcid":false,"given":"David","family":"M\u00fcnch","sequence":"additional","affiliation":[{"name":"Fraunhofer IOSB, Gutleuthausstra\u00dfe 1, 76275 Ettlingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7857-0332","authenticated-orcid":false,"given":"Michael","family":"Arens","sequence":"additional","affiliation":[{"name":"Fraunhofer IOSB, Gutleuthausstra\u00dfe 1, 76275 Ettlingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,8]]},"reference":[{"key":"ref_1","unstructured":"Das, A., and Rad, P. (2020). Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Chen, Y., Ouyang, L., Bao, F.S., Li, Q., Han, L., Zhu, B., Ge, Y., Robinson, P., Xu, M., and Liu, J. (2020). An interpretable machine learning framework for accurate severe vs. non-severe COVID-19 clinical type classification. Lancet.","DOI":"10.2139\/ssrn.3638427"},{"key":"ref_3","unstructured":"Fan, X., Liu, S., Chen, J., and Henderson, T.C. (2020). An investigation of COVID-19 spreading factors with explainable ai techniques. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Karim, M., D\u00f6hmen, T., Rebholz-Schuhmann, D., Decker, S., Cochez, M., and Beyan, O. (2020). Deepcovidexplainer: Explainable COVID-19 predictions based on chest X-ray images. arXiv.","DOI":"10.1109\/BIBM49941.2020.9313304"},{"key":"ref_5","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv."},{"key":"ref_6","unstructured":"Goodfellow, I.J., Shlens, J., and Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). Why should i trust you? Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Moosavi-Dezfooli, S.M. (2016, January 27\u201330). DeepFool: A simple and accurate method to fool deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.282"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bose, A.J., and Aarabi, P. (2018, January 29\u201331). Adversarial attacks on face detectors using neural net based constrained optimization. Proceedings of the IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), Vancouver, BC, Canada.","DOI":"10.1109\/MMSP.2018.8547128"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Jia, R., and Liang, P. (2017). Adversarial examples for evaluating reading comprehension systems. arXiv.","DOI":"10.18653\/v1\/D17-1215"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., and Song, D. (2018, January 18\u201323). Robust physical-world attacks on deep learning visual classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00175"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ranjan, A., Janai, J., Geiger, A., and Black, M.J. (2019, January 27\u201328). Attacking Optical Flow. Proceedings of the International Conference on Computer Vision, Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00249"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sharif, M., Bhagavatula, S., Bauer, L., and Reiter, M.K. (2016, January 24\u201328). Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria.","DOI":"10.1145\/2976749.2978392"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3317611","article-title":"A General Framework for Adversarial Examples with Objectives","volume":"22","author":"Sharif","year":"2019","journal-title":"ACM Trans. Priv. Secur. (TOPS)"},{"key":"ref_15","unstructured":"Goodman, F. (2016, January 23). European union regulations on algorithmic decision-making and a right to explanation. Proceedings of the ICML Workshop on Human Interpretability in ML, New York, NY, USA."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1109\/MC.2021.3074263","article-title":"The Ten Commandments of ethical medical AI","volume":"54","author":"Muller","year":"2021","journal-title":"Computer"},{"key":"ref_17","unstructured":"Holmes, J., and Meyerhoff, M. (2008). The Handbook of Language and Gender, John Wiley & Sons."},{"key":"ref_18","first-page":"4349","article-title":"Man is to computer programmer as woman is to homemaker? debiasing word embeddings","volume":"29","author":"Bolukbasi","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1126\/science.aaa8685","article-title":"Advances in natural language processing","volume":"349","author":"Hirschberg","year":"2015","journal-title":"Science"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1464","DOI":"10.1037\/0022-3514.74.6.1464","article-title":"Measuring individual differences in implicit cognition: The implicit association test","volume":"74","author":"Greenwald","year":"1998","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_21","unstructured":"Chakraborty, T., Badie, G., and Rudder, B. (2018, March 15). Reducing Gender Bias in Word Embeddings. Computer Science Department, Stanford University. Available online: http:\/\/cs229.stanford.edu\/proj2016\/report\/."},{"key":"ref_22","unstructured":"Font, J.E., and Costa-Jussa, M.R. (2019). Equalizing gender biases in neural machine translation with word embeddings techniques. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Nguyen, A., Yosinski, J., and Clune, J. (2015, January 7\u201312). Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298640"},{"key":"ref_24","unstructured":"Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., and Brendel, W. (2018). ImageNet-trained CNNs are biased towards texture; Increasing shape bias improves accuracy and robustness. arXiv."},{"key":"ref_25","first-page":"297","article-title":"Generalized additive model (GAM)","volume":"1","author":"Hastie","year":"1986","journal-title":"Stat. Sci."},{"key":"ref_26","unstructured":"Breiman, L., Friedman, J., Olshen, R., and Stone, C. (1984). Classification and Regression Trees. Monterey, Wadsworth International Group."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Friedman, J., Hastie, T., and Tibshirani, R. (2001). The Elements of Statistical Learning, Springer.","DOI":"10.1007\/978-0-387-21606-5"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Salzberg, S.L. (1994). C4. 5: Programs for Machine Learning by J. Ross Quinlan. Morgan Kaufmann Publishers, Inc., 1993, Kluwer Academic.","DOI":"10.1007\/BF00993309"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1080\/10618600.2014.907095","article-title":"Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation","volume":"24","author":"Goldstein","year":"2015","journal-title":"J. Comput. Graph. Stat."},{"key":"ref_31","unstructured":"Van Lent, M., Fisher, W., and Mancuso, M. (2004, January 25\u201329). An explainable artificial intelligence system for small-unit tactical behavior. Proceedings of the National Conference on Artificial Intelligence, San Jose, CA, USA."},{"key":"ref_32","unstructured":"Shortliffe, E.H. (1997, January 5). Mycin: A knowledge-based computer program applied to infectious diseases. Proceedings of the Annual Symposium on Computer Application in Medical Care, San Francisco, CA, USA."},{"key":"ref_33","first-page":"1803","article-title":"How to explain individual classification decisions","volume":"11","author":"Baehrens","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1080\/08839519308949993","article-title":"Inductive and Bayesian learning in medical diagnosis","volume":"7","author":"Kononenko","year":"1993","journal-title":"Appl. Artif. Intell. Int. J."},{"key":"ref_36","unstructured":"Becker, B., Kohavi, R., and Sommerfield, D. (2001). Visualizing the simple Bayesian classifier. Information Visualization in Data Mining and Knowledge Discovery, Morgan Kaufmann Publishers Inc.. Available online: https:\/\/dl.acm.org\/doi\/10.5555\/383784.383809."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Mo\u017eina, M., Dem\u0161ar, J., Kattan, M., and Zupan, B. (2004). Nomograms for visualization of naive Bayesian classifier. European Conference on Principles of Data Mining and Knowledge Discovery, Proceedings of the 8th European Conference on Principles and Practice of Knowledge Discovery in Databases, Pisa, Italy, 20\u201324 September 2004, Springer.","DOI":"10.1007\/978-3-540-30116-5_32"},{"key":"ref_38","unstructured":"Poulet, F. (2004, January 1\u20134). Svm and graphical algorithms: A cooperative approach. Proceedings of the Fourth IEEE International Conference on Data Mining (ICDM\u201904), Brighton, UK."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Hamel, L. (2006, January 28\u201329). Visualization of support vector machines with unsupervised learning. Proceedings of the IEEE Symposium on Computational Intelligence and Bioinformatics and Computational Biology, Toronto, ON, Canada.","DOI":"10.1109\/CIBCB.2006.330984"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_41","first-page":"2843","article-title":"Deep neural networks segment neuronal membranes in electron microscopy images","volume":"25","author":"Ciresan","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ciresan, D., Meier, U., and Schmidhuber, J. (2012, January 16\u201321). Multi-column deep neural networks for image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6248110"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Le, Q.V., Zou, W.Y., Yeung, S.Y., and Ng, A.Y. (2011, January 20\u201325). Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995496"},{"key":"ref_44","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"ref_46","first-page":"2493","article-title":"Natural language processing (almost) from scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3236009","article-title":"A survey of methods for explaining black box models","volume":"51","author":"Guidotti","year":"2019","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Goebel, R., Chander, A., Holzinger, K., Lecue, F., Akata, Z., Stumpf, S., Kieseberg, P., and Holzinger, A. (2018). Explainable AI: The new 42?. International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Proceedings of the Second IFIP TC 5, TC 8\/WG 8.4, 8.9, TC 12\/WG 12.9 International Cross-Domain Conference, CD-MAKE 2018, Hamburg, Germany, 27\u201330 August 2018, Springer.","DOI":"10.1007\/978-3-319-99740-7_21"},{"key":"ref_50","unstructured":"Lipton, Z.C. (2016). The mythos of model interpretability. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014). Visualizing and Understanding Convolutional Networks, Springer.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Mahendran, A., and Vedaldi, A. (2015, January 7\u201312). Understanding deep image representations by inverting them. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299155"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layerwise relevance propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.patcog.2016.11.008","article-title":"Explaining nonlinear classification decisions with deep taylor decomposition","volume":"65","author":"Montavon","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_55","unstructured":"Springenberg, J., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015, January 7\u20139). Striving for Simplicity: The All Convolutional Net. Proceedings of the ICLR (Workshop Track), San Diego, CA, USA."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Zhang, R., Isola, P., and Efros, A.A. (2016). Colorful image colorization. European Conference on Computer Vision (ECCV), Proceedings of the 14th European Conference, Amsterdam, The Netherlands, 11\u201314 October 2016, Springer.","DOI":"10.1007\/978-3-319-46487-9_40"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1084","DOI":"10.1007\/s11263-017-1059-x","article-title":"Top-down neural attention by excitation backprop","volume":"126","author":"Zhang","year":"2018","journal-title":"Int. J. Comput. Vis."},{"key":"ref_58","first-page":"3387","article-title":"Synthesizing the preferred inputs for neurons in neural networks via deep generator networks","volume":"29","author":"Nguyen","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_59","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"2660","DOI":"10.1109\/TNNLS.2016.2599820","article-title":"Evaluating the visualization of what a deep neural network has learned","volume":"28","author":"Samek","year":"2016","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"ref_61","unstructured":"Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., and Lipson, H. (2015). Understanding neural networks through deep visualization. arXiv."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Dosovitskiy, A., and Brox, T. (2016, January 27\u201330). Inverting visual representations with convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.522"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"e15","DOI":"10.23915\/distill.00015","article-title":"Activation Atlas","volume":"4","author":"Carter","year":"2019","journal-title":"Distill"},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1\u20133). Explaining explanations: An overview of interpretability of machine learning. Proceedings of the IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"ref_65","unstructured":"Remler, D.K., and Van Ryzin, G.G. (2014). Research Methods in Practice: Strategies for Description and Causation, Sage Publications."},{"key":"ref_66","unstructured":"Ridgeway, G., Madigan, D., Richardson, T., and O\u2019Kane, J. (1998). Interpretable Boosted Na\u00efve Bayes Classification, AAAI."},{"key":"ref_67","unstructured":"Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.inffus.2021.01.008","article-title":"Towards multi-modal causability with Graph Neural Networks enabling information fusion for explainable AI","volume":"71","author":"Holzinger","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"782","DOI":"10.1016\/j.dss.2011.01.013","article-title":"Performance of classification models from a user perspective","volume":"51","author":"Martens","year":"2011","journal-title":"Decis. Support Syst."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2594473.2594475","article-title":"Comprehensible classification models: A position paper","volume":"15","author":"Freitas","year":"2014","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"1350","DOI":"10.1214\/15-AOAS848","article-title":"Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model","volume":"9","author":"Letham","year":"2015","journal-title":"Ann. Appl. Stat."},{"key":"ref_72","unstructured":"Ustun, B., and Rudin, C. (2014). Methods and models for interpretable linear classification. arXiv."},{"key":"ref_73","unstructured":"Letham, B., Rudin, C., McCormick, T.H., and Madigan, D. (2013, January 14\u201318). An interpretable stroke prediction model using rules and Bayesian analysis. Proceedings of the Workshops at the Twenty-Seventh AAAI Conference on Artificial Intelligence, Bellevue, WA, USA."},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Zilke, J.R., Menc\u00eda, E.L., and Janssen, F. (2016). DeepRED\u2014Rule extraction from deep neural networks. International Conference on Discovery Science, Proceedings of the 19th International Conference, DS 2016, Bari, Italy, 19\u201321 October 2016, Springer.","DOI":"10.1007\/978-3-319-46307-0_29"},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"1114","DOI":"10.1109\/21.299696","article-title":"Rule generation from neural networks","volume":"24","author":"Fu","year":"1994","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_76","first-page":"24","article-title":"Extracting tree-structured representations of trained networks","volume":"8","author":"Craven","year":"1996","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_77","unstructured":"Bottou, L., Peters, J., Qui\u00f1onero-Candela, J., Charles, D.X., Chickering, D.M., Portugaly, E., Ray, D., Simard, P., and Snelson, E. (2013). Counterfactual reasoning and learning systems: The example of computational advertising. J. Mach. Learn. Res., 14."},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1093\/pan\/mpt019","article-title":"Kernel regularized least squares: Reducing misspecification bias with a flexible and interpretable machine learning approach","volume":"22","author":"Hainmueller","year":"2014","journal-title":"Politi. Anal."},{"key":"ref_79","unstructured":"Nie, W., Zhang, Y., and Patel, A. (2018, January 10\u201315). A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_80","unstructured":"Kindermans, P.J., Sch\u00fctt, K.T., Alber, M., M\u00fcller, K.R., Erhan, D., Kim, B., and D\u00e4hne, S. (2017). Learning how to explain neural networks: Patternnet and patternattribution. arXiv."},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.neuroimage.2013.10.067","article-title":"On the interpretation of weight vectors of linear models in multivariate neuroimaging","volume":"87","author":"Haufe","year":"2014","journal-title":"Neuroimage"},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Lakkaraju, H., Bach, S.H., and Leskovec, J. (2016, January 13\u201317). Interpretable decision sets: A joint framework for description and prediction. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939874"},{"key":"ref_83","unstructured":"Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., and Stewart, W. (2016, January 5\u201310). Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_84","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017, January 6\u201311). Learning important features through propagating activation differences. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27\u201330). Learning deep features for discriminative localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.319"},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Chattopadhyay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. (2018, January 12\u201315). Grad-CAM++: Generalized Gradient-based Visual Explanations for Deep Convolutional Networks. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.","DOI":"10.1109\/WACV.2018.00097"},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_89","unstructured":"(2020, January 26\u201330). On Evaluating Explainability Algorithms. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_90","unstructured":"Babiker, H.K.B., and Goebel, R. (2017). An introduction to deep visual explanation. arXiv."},{"key":"ref_91","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014). Microsoft coco: Common objects in context. European Conference on Computer Vision, Proceedings of the 13th European Conference, Zurich, Switzerland, 6\u201312 September 2014, Springer.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_92","unstructured":"Babiker, H.K.B., and Goebel, R. (2017). Using KL-divergence to focus deep visual explanation. arXiv."},{"key":"ref_93","unstructured":"Zintgraf, L.M., Cohen, T.S., Adel, T., and Welling, M. (2017). Visualizing deep neural network decisions: Prediction difference analysis. arXiv."},{"key":"ref_94","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1109\/TKDE.2007.190734","article-title":"Explaining classifications for individual instances","volume":"20","author":"Kononenko","year":"2008","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_95","unstructured":"Smilkov, D., Thorat, N., Kim, B., Vi\u00e9gas, F., and Wattenberg, M. (2017). Smoothgrad: Removing noise by adding noise. arXiv."},{"key":"ref_96","unstructured":"Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6\u201311). Axiomatic attribution for deep networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_97","doi-asserted-by":"crossref","unstructured":"Huk Park, D., Anne Hendricks, L., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., and Rohrbach, M. (2018, January 18\u201323). Multimodal explanations: Justifying decisions and pointing to the evidence. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00915"},{"key":"ref_98","doi-asserted-by":"crossref","unstructured":"Hohman, F., Park, H., Robinson, C., and Chau, D.H. (2019). Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations. arXiv.","DOI":"10.1109\/TVCG.2019.2934659"},{"key":"ref_99","doi-asserted-by":"crossref","unstructured":"Buhrmester, V., M\u00fcnch, D., Bulatov, D., and Arens, M. (2019). Evaluating the Impact of Color Information in Deep Neural Networks. Iberian Conference on Pattern Recognition and Image Analysis, Proceedings of the 9th Iberian Conference, IbPRIA 2019, Madrid, Spain, 1\u20134 July 2019, Springer.","DOI":"10.1007\/978-3-030-31332-6_27"},{"key":"ref_100","unstructured":"Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6\u201311). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the International Conference on Machine Learning, Lille, France."},{"key":"ref_101","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.jneumeth.2016.10.008","article-title":"Interpretable deep neural networks for single-trial EEG classification","volume":"274","author":"Sturm","year":"2016","journal-title":"J. Neurosci. Methods"},{"key":"ref_102","unstructured":"Bojarski, M., Choromanska, A., Choromanski, K., Firner, B., Jackel, L., Muller, U., and Zieba, K. (2016). Visualbackprop: Visualizing cnns for autonomous driving. arXiv."},{"key":"ref_103","doi-asserted-by":"crossref","unstructured":"Fong, R.C., and Vedaldi, A. (2017, January 22\u201329). Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.371"},{"key":"ref_104","doi-asserted-by":"crossref","unstructured":"Lei, T., Barzilay, R., and Jaakkola, T. (2016). Rationalizing neural predictions. arXiv.","DOI":"10.18653\/v1\/D16-1011"},{"key":"ref_105","unstructured":"Radford, A., Jozefowicz, R., and Sutskever, I. (2017). Learning to generate reviews and discovering sentiment. arXiv."},{"key":"ref_106","unstructured":"Thiagarajan, J.J., Kailkhura, B., Sattigeri, P., and Ramamurthy, K.N. (2016). TreeView: Peeking into deep neural networks via feature-space partitioning. arXiv."},{"key":"ref_107","unstructured":"Shwartz-Ziv, R., and Tishby, N. (2017). Opening the black box of deep neural networks via information. arXiv."},{"key":"ref_108","doi-asserted-by":"crossref","unstructured":"Turner, R. (2016, January 13\u201316). A model explanation system. Proceedings of the IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), Vietri sul Mare, Italy.","DOI":"10.1109\/MLSP.2016.7738872"},{"key":"ref_109","doi-asserted-by":"crossref","unstructured":"Krishnan, S., and Wu, E. (2017, January 14\u201319). Palm: Machine learning explanations for iterative debugging. Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics, Chicago, IL, USA.","DOI":"10.1145\/3077257.3077271"},{"key":"ref_110","unstructured":"Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., and Doshi-Velez, F. (2018). How do humans understand explanations from machine learning systems? An evaluation of the human-interpretability of explanation. arXiv."},{"key":"ref_111","doi-asserted-by":"crossref","unstructured":"El Bekri, N., Kling, J., and Huber, M.F. (2019, January 13\u201315). A Study on Trust in Black Box Models and Post hoc Explanations. Proceedings of the 14th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2019), Seville, Spain.","DOI":"10.1007\/978-3-030-20055-8_4"},{"key":"ref_112","doi-asserted-by":"crossref","unstructured":"Fong, R., Patrick, M., and Vedaldi, A. (2019). Understanding Deep Networks via Extremal Perturbations and Smooth Masks. arXiv.","DOI":"10.1109\/ICCV.2019.00304"},{"key":"ref_113","unstructured":"(2021, October 07). Torchray. Available online: github.com\/facebookresearch\/TorchRay."},{"key":"ref_114","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1109\/TRPMS.2021.3066428","article-title":"On interpretability of artificial neural networks: A survey","volume":"5","author":"Fan","year":"2021","journal-title":"IEEE Trans. Radiat. Plasma Med. Sci."},{"key":"ref_115","unstructured":"Fan, F.L., Xiong, J., Li, M., and Wang, G. (2021, October 07). IndependentEvaluation GitHub Code. Available online: https:\/\/github.com\/FengleiFan\/IndependentEvaluation."},{"key":"ref_116","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1613\/jair.1.12228","article-title":"A Survey on the Explainability of Supervised Machine Learning","volume":"70","author":"Burkart","year":"2021","journal-title":"J. Artif. Intell. Res."},{"key":"ref_117","doi-asserted-by":"crossref","unstructured":"Agarwal, N., and Das, S. (2020, January 1\u20134). Interpretable Machine Learning Tools: A Survey. Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI), Canberra, Australia.","DOI":"10.1109\/SSCI47803.2020.9308260"},{"key":"ref_118","unstructured":"Hooker, S., Erhan, D., Kindermans, P.J., and Kim, B. (2018). Evaluating feature importance estimates. arXiv."},{"key":"ref_119","unstructured":"Yeh, C.K., Hsieh, C.Y., Suggala, A.S., Inouye, D., and Ravikumar, P. (2019). How Sensitive are Sensitivity-Based Explanations?. arXiv."},{"key":"ref_120","doi-asserted-by":"crossref","unstructured":"Alexandrov, N. (2017, January 5\u20139). Explainable AI decisions for human-autonomy interactions. Proceedings of the 17th AIAA Aviation Technology, Integration, and Operations Conference, Denver, CO, USA.","DOI":"10.2514\/6.2017-3991"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/4\/48\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:43:22Z","timestamp":1760168602000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/4\/48"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,8]]},"references-count":120,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["make3040048"],"URL":"https:\/\/doi.org\/10.3390\/make3040048","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,8]]}}}