{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T18:32:46Z","timestamp":1778610766804,"version":"3.51.4"},"reference-count":36,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2023,10,23]],"date-time":"2023-10-23T00:00:00Z","timestamp":1698019200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Axioms"],"abstract":"<jats:p>In the explainable artificial intelligence (XAI) field, an algorithm or a tool can help people understand how a model makes a decision. And this can help to select important features to reduce computational costs to realize high-performance computing. But existing methods are usually used to visualize important features or highlight active neurons, and few of them show the importance of relationships between features. In recent years, some methods based on a white-box approach have taken relationships between features into account, but most of them can only work on some specific models. Although methods based on a black-box approach can solve the above problems, most of them can only be applied to tabular data or text data instead of image data. To solve these problems, we propose a local interpretable model-agnostic explanation approach based on feature relationships. This approach combines the relationships between features into the interpretation process and then visualizes the interpretation results. Finally, this paper conducts a lot of experiments to evaluate the correctness of relationships between features and evaluates this XAI method in terms of accuracy, fidelity, and consistency.<\/jats:p>","DOI":"10.3390\/axioms12100997","type":"journal-article","created":{"date-parts":[[2023,10,23]],"date-time":"2023-10-23T01:41:21Z","timestamp":1698025281000},"page":"997","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Interpretable Model-Agnostic Explanations Based on Feature Relationships for High-Performance Computing"],"prefix":"10.3390","volume":"12","author":[{"given":"Zhouyuan","family":"Chen","sequence":"first","affiliation":[{"name":"School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing 214400, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhichao","family":"Lian","sequence":"additional","affiliation":[{"name":"School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing 214400, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhe","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing 214400, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,10,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_2","first-page":"180","article-title":"Retracted Article: Based on deep learning in traffic remote sensing image processing to recognize target vehicle","volume":"45","author":"Wang","year":"2023","journal-title":"Int. J. Comput. Appl."},{"key":"ref_3","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Yuan, Y., Zhou, X., Pan, S., Zhu, Q., Song, Z., and Guo, L. (2021, January 19\u201327). A Relation-Specific Attention Network for Joint Entity and Relation Extraction. Proceedings of the International Joint Conference on Artificial Intelligence, Montreal, QC, Canada.","DOI":"10.24963\/ijcai.2020\/561"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Graves, A., Mohamed, A.R., and Hinton, G. (2013, January 26\u201331). Speech recognition with deep recurrent neural networks. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6638947"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"61672","DOI":"10.1109\/ACCESS.2020.2984368","article-title":"Multimodal approach of speech emotion recognition using multi-level multi-head fusion attention-based recurrent neural network","volume":"8","author":"Ho","year":"2020","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Li, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., and Xia, J. (2020). Regarding artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT. Radiology, 201178.","DOI":"10.1148\/radiol.2020200905"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","article-title":"A survey on explainable artificial intelligence (xai): Toward medical xai","volume":"32","author":"Tjoa","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"370","DOI":"10.2307\/2344614","article-title":"Generalized linear models","volume":"135","author":"Nelder","year":"1972","journal-title":"J. R. Stat. Soc. Ser. A Stat. Soc."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Yang, Y., Ma, H., and Wu, Y.N. (2019, January 15\u201320). Interpreting cnns via decision trees. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00642"},{"key":"ref_11","unstructured":"Wan, A., Dunlap, L., Ho, D., Yin, J., Lee, S., Jin, H., and Gonzalez, J.E. (2020). NBDT: Neural-backed decision trees. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). Why should i trust you? Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_13","first-page":"4768","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Pawelczyk, M., Broelemann, K., and Kasneci, G. (2020, January 20\u201324). Learning model-agnostic counterfactual explanations for tabular data. Proceedings of the Web Conference 2020, Taipei, Taiwan.","DOI":"10.1145\/3366423.3380087"},{"key":"ref_15","unstructured":"Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016, January 5\u201310). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_16","unstructured":"Nguyen, A., Yosinski, J., and Clune, J. (2016). Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., and Yuan, C. (2022, January 18\u201324). Focal and global knowledge distillation for detectors. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00460"},{"key":"ref_18","first-page":"16468","article-title":"Instance-conditional knowledge distillation for object detection","volume":"34","author":"Kang","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Iwana, B.K., Kuroki, R., and Uchida, S. (2019, January 27\u201328). Explaining convolutional neural networks using softmax gradient layer-wise relevance propagation. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision Workshop, Seoul, Republic of Korea.","DOI":"10.1109\/ICCVW.2019.00513"},{"key":"ref_21","unstructured":"Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., and Viegas, F. (2018, January 10\u201315). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"8799","DOI":"10.1109\/TKDE.2022.3209997","article-title":"Concept-Level Model Interpretation From the Causal Aspect","volume":"35","author":"Yao","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mahendran, A., and Vedaldi, A. (2015, January 7\u201312). Understanding deep image representations by inverting them. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299155"},{"key":"ref_24","unstructured":"Kim, B., Khanna, R., and Koyejo, O.O. (2016, January 5\u201310). Examples are not enough, learn to criticize! criticism for interpretability. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Cao, R., Shi, F., Wu, Y.N., and Zhu, S.C. (2018, January 2\u20137). Interpreting CNN knowledge via an explanatory graph. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11819"},{"key":"ref_26","unstructured":"Dikopoulou, Z., Moustakidis, S., and Karlsson, P. (2017). GLIME: A new graphical methodology for interpretable model-agnostic explanations. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"103502","DOI":"10.1016\/j.artint.2021.103502","article-title":"Explaining individual predictions when features are dependent: More accurate approximations to Shapley values","volume":"298","author":"Aas","year":"2021","journal-title":"Artif. Intell."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2018, January 2\u20137). Anchors: High-precision model-agnostic explanations. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11491"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Messalas, A., Aridas, C., and Kanellopoulos, Y. (2020, January 19\u201322). Evaluating MASHAP as a faster alternative to LIME for model-agnostic machine learning interpretability. Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Busan, Republic of Korea.","DOI":"10.1109\/BigData50022.2020.9378034"},{"key":"ref_31","unstructured":"Wang, A., and Lee, W.N. (2022). Exploring Concept Contribution Spatially: Hidden Layer Interpretation with Spatial Activation Concept Vector. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ge, Y., Xiao, Y., Xu, Z., Zheng, M., Karanam, S., Chen, T., and Wu, Z. (2021, January 20\u201325). A peek into the reasoning of neural networks: Interpreting with structural visual concepts. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00223"},{"key":"ref_33","first-page":"6021","article-title":"Sanity checks for saliency metrics","volume":"34","author":"Tomsett","year":"2020","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_34","first-page":"7786","article-title":"Towards robust interpretability with self-explaining neural networks","volume":"31","author":"Jaakkola","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","article-title":"Methods for interpreting and understanding deep neural networks","volume":"73","author":"Montavon","year":"2018","journal-title":"Digit. Signal Process."},{"key":"ref_36","unstructured":"Petsiuk, V., Das, A., and Saenko, K. (2018). Rise: Randomized input sampling for explanation of black-box models. arXiv."}],"container-title":["Axioms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-1680\/12\/10\/997\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:10:03Z","timestamp":1760130603000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-1680\/12\/10\/997"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,23]]},"references-count":36,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["axioms12100997"],"URL":"https:\/\/doi.org\/10.3390\/axioms12100997","relation":{},"ISSN":["2075-1680"],"issn-type":[{"value":"2075-1680","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,23]]}}}