{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T17:28:08Z","timestamp":1764955688088,"version":"3.46.0"},"reference-count":48,"publisher":"Walter de Gruyter GmbH","issue":"1","license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,4,26]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>A good AI algorithm can make accurate predictions and provide reasonable explanations for the field in which it is applied. However, the application of deep models makes the black box problem, i.e., the lack of interpretability of a model, more prominent. In particular, when there are multiple features in an application domain and complex interactions between these features, it is difficult for a deep model to intuitively explain its prediction results. Moreover, in practical applications, multiorder feature interactions are ubiquitous. To break the interpretation limitations of deep models, we argue that a multiorder linearly separable deep model can be divided into different orders to explain its prediction results. Inspired by the interpretability advantage of tree models, we design a feature representation mechanism that can consistently represent the features of both trees and deep models. Based on the consistent representation, we propose a multiorder feature-tracking strategy to provide a prediction-oriented multiorder explanation for a linearly separable deep model. In experiments, we have empirically verified the effectiveness of our approach in two binary classification application scenarios: education and marketing. Experimental results show that our model can intuitively represent complex relationships between features through diversified multiorder explanations.<\/jats:p>","DOI":"10.1515\/jisys-2022-0212","type":"journal-article","created":{"date-parts":[[2023,4,28]],"date-time":"2023-04-28T11:06:39Z","timestamp":1682679999000},"source":"Crossref","is-referenced-by-count":0,"title":["A multiorder feature tracking and explanation strategy for explainable deep learning"],"prefix":"10.1515","volume":"32","author":[{"given":"Lin","family":"Zheng","sequence":"first","affiliation":[{"name":"Department of Computer Science, College of Engineering, Shantou University , Shantou 515063 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yixuan","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Computer Science, College of Engineering, Shantou University , Shantou 515063 , China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"374","published-online":{"date-parts":[[2023,4,26]]},"reference":[{"key":"2025120517213984434_j_jisys-2022-0212_ref_001","doi-asserted-by":"crossref","unstructured":"Zheng L, Zhu F, Huang S, Xie J. Context neighbor recommender: integrating contexts via neighbors for recommendations. Inform Sci. 2017;414(11):1\u201318. http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0020025517307466.","DOI":"10.1016\/j.ins.2017.05.034"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_002","doi-asserted-by":"crossref","unstructured":"Zheng L, Guo N, Chen W, Yu J, Jiang D. Sentiment-guided sequential recommendation. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval; 2020. p. 1957\u201360.","DOI":"10.1145\/3397271.3401330"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_003","doi-asserted-by":"crossref","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D. A survey of methods for explaining black box models. ACM computing surveys (CSUR). 2018;51(5):1\u201342.","DOI":"10.1145\/3236009"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_004","doi-asserted-by":"crossref","unstructured":"Lipton ZC. The Mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue. 2018;16(3):31\u201357.","DOI":"10.1145\/3236386.3241340"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_005","unstructured":"Molnar C. Interpretable machine learning. Lulu. com; 2020."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_006","doi-asserted-by":"crossref","unstructured":"Guidotti R, Monreale A, Matwin S, Pedreschi D. Black box explanation by learning image exemplars in the latent feature space. 2020. arXiv: http:\/\/arXiv.org\/abs\/arXiv:200203746.","DOI":"10.1007\/978-3-030-46150-8_12"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_007","doi-asserted-by":"crossref","unstructured":"Bhatt U, Weller A, Moura JM. Evaluating and aggregating feature-based model explanations. 2020. arXiv: http:\/\/arXiv.org\/abs\/arXiv:200500631.","DOI":"10.24963\/ijcai.2020\/417"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_008","unstructured":"Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: Proceedings of the 31st international conference on neural information processing systems; 2017. p. 4768\u201377."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_009","unstructured":"Lundberg SM, Erion GG, Lee SI. Consistent individualized feature attribution for tree ensembles. 2018. arXiv: http:\/\/arXiv.org\/abs\/arXiv:180203888."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_010","doi-asserted-by":"crossref","unstructured":"Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, et al. From local explanations to global understanding with explainable AI for trees. Nature Machine Intell. 2020;2(1):56\u201367.","DOI":"10.1038\/s42256-019-0138-9"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_011","doi-asserted-by":"crossref","unstructured":"Sharma P, Mirzan SR, Bhandari A, Pimpley A, Eswaran A, Srinivasan S, et al. Evaluating tree explanation methods for anomaly reasoning: a case study of SHAP TreeExplainer and TreeInterpreter. In: International Conference on Conceptual Modeling. Springer; 2020. p. 35\u201345.","DOI":"10.1007\/978-3-030-65847-2_4"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_012","doi-asserted-by":"crossref","unstructured":"Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat. 2001;29(5):1189\u2013232. http:\/\/www.jstor.org\/stable\/2699986.","DOI":"10.1214\/aos\/1013203451"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_013","doi-asserted-by":"crossref","unstructured":"Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD \u201916. New York, NY, USA: Association for Computing Machinery; 2016. p. 785\u201394. 10.1145\/2939672.2939785.","DOI":"10.1145\/2939672.2939785"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_014","doi-asserted-by":"crossref","unstructured":"Stoji\u0107 A, Stani\u0107 N, Vukovi\u0107 G, Stani\u0161i\u0107 S, Peri\u0161i\u0107 M, \u0160o\u0161tari\u0107 A, et al. Explainable extreme gradient boosting tree-based prediction of toluene, ethylbenzene and xylene wet deposition. Sci Total Environ. 2019;653:140\u20137.","DOI":"10.1016\/j.scitotenv.2018.10.368"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_015","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez JAF. United States banking stability: An explanation through machine learning. Banks Bank Syst. 2020;15(4):137.","DOI":"10.21511\/bbs.15(4).2020.12"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_016","doi-asserted-by":"crossref","unstructured":"He X, Pan J, Jin O, Xu T, Liu B, Xu T, et al. Practical lessons from predicting clicks on ads at Facebook. In: Proceedings of the Eighth International Workshop on Data Mining for Online Advertising. ADKDD\u201914. New York, NY, USA: Association for Computing Machinery; 2014. p. 1\u20139. 10.1145\/2648584.2648589.","DOI":"10.1145\/2648584.2648589"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_017","unstructured":"Zheng L, Zhu F, Mohammed A. Attribute and global boosting: a rating prediction method in context-aware recommendation. Comput J. 2017;60(7):957\u201368. https:\/\/academic.oup.com\/comjnl\/article\/60\/7\/957\/2609377."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_018","doi-asserted-by":"crossref","unstructured":"Shih A, Choi A, Darwiche A. A symbolic approach to explaining Bayesian network classifiers. 2018. arXiv: http:\/\/arXiv.org\/abs\/arXiv:180503364.","DOI":"10.24963\/ijcai.2018\/708"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_019","unstructured":"Flambeau JKF, Norbert T. Simplifying the explanation of deep neural networks with sufficient and necessary feature-sets: case of text classification. 2020. arXiv: http:\/\/arXiv.org\/abs\/arXiv:201003724."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_020","doi-asserted-by":"crossref","unstructured":"He X, Chua TS. Neural factorization machines for sparse predictive analytics. In: Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval; 2017. p. 355\u201364.","DOI":"10.1145\/3077136.3080777"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_021","doi-asserted-by":"crossref","unstructured":"Xiao J, Ye H, He X, Zhang H, Wu F, Chua TS. Attentional factorization machines: Learning the weight of feature interactions via attention networks. 2017. arXiv: http:\/\/arXiv.org\/abs\/arXiv:170804617.","DOI":"10.24963\/ijcai.2017\/435"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_022","doi-asserted-by":"crossref","unstructured":"Rendle S. Factorization machines with libfm. ACM Trans Intell Syst Technol (TIST). 2012;3(3):1\u201322.","DOI":"10.1145\/2168752.2168771"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_023","unstructured":"Shrikumar A, Greenside P, Kundaje A. Learning important features through propagating activation differences. In: International Conference on Machine Learning. PMLR; 2017. p. 3145\u201353."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_024","unstructured":"Zhang Y, Tinnno P, Leonardis A, Tang K. A survey on neural network interpretability. 2020. arXiv: http:\/\/arXiv.org\/abs\/arXiv:201214261."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_025","doi-asserted-by":"crossref","unstructured":"Zilke JR, Menciiia EL, Janssen F. Deepred-rule extraction from deep neural networks. In: International Conference on Discovery Science. Springer; 2016. p. 457\u201373.","DOI":"10.1007\/978-3-319-46307-0_29"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_026","unstructured":"Wan A, Dunlap L, Ho D, Yin J, Lee S, Jin H, et al. NBDT: Neural-backed decision trees. 2020. arXiv: http:\/\/arXiv.org\/abs\/arXiv:200400221."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_027","doi-asserted-by":"crossref","unstructured":"Wang X, He X, Feng F, Nie L, Chua TS. TEM: Tree-enhanced embedding model for explainable recommendation. In: Proceedings of the 2018 Conference. WWW \u201918. Republic and Canton of Geneva, CHE: International Conferences Steering Committee; 2018. p. 1543\u201352. 10.1145\/3178876.3186066.","DOI":"10.1145\/3178876.3186066"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_028","doi-asserted-by":"crossref","unstructured":"Humbird KD, Peterson JL, McClarren RG. Deep neural network initialization with decision trees. IEEE Trans Neural Networks Learn Syst. 2018;30(5):1286\u201395.","DOI":"10.1109\/TNNLS.2018.2869694"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_029","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems. Red Hook, NY, United States: Curran Associates Inc.; 2013. p. 3111\u20139."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_030","doi-asserted-by":"crossref","unstructured":"Bayer I, He X, Kanagal B, Rendle S. A Generic Coordinate Descent Framework for Learning from Implicit Feedback. In: Proceedings of the 26th International Conference on WWW \u201917. Republic and Canton of Geneva, CHE: International Conferences Steering Committee; 2017. p. 1341\u201350. 10.1145\/3038912.3052694.","DOI":"10.1145\/3038912.3052694"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_031","unstructured":"Socher R, Chen D, Manning CD, Ng A. Reasoning with neural tensor networks for knowledge base completion. In: Advances in neural information processing systems. Red Hook, NY, United States: Curran Associates Inc.; 2013. p. 926\u201334."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_032","unstructured":"Chen D, Socher R, Manning CD, Ng AY. Learning new facts from knowledge bases with neural tensor networks and semantic word vectors. 2013. arXiv: http:\/\/arXiv.org\/abs\/arXiv:13013618."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_033","unstructured":"Weston J, Chopra S, Bordes A. Memory networks. In: Proceedings of the International Conference on Learning Representations. ICLR \u201915; 2015."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_034","unstructured":"Sukhbaatar S, Szlam A, Weston J, Fergus R. End-to-end memory networks. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in neural information processing systems 28. NIPS \u201915. Cambridge, MA, United States: MIT Press; 2015. p. 2440\u20138."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_035","unstructured":"Blondel M, Fujino A, Ueda N, Ishihata M. Higher-order factorization machines. 2016. arXiv: http:\/\/arXiv.org\/abs\/arXiv:160707195."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_036","doi-asserted-by":"crossref","unstructured":"Bach S, Binder A, Montavon G, Klauschen F, M\u00fcller KR, Samek W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One. 2015;10(7):e0130140.","DOI":"10.1371\/journal.pone.0130140"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_037","unstructured":"Balduzzi D, McWilliams B, Butler-Yeoman T. Neural Taylor approximations: convergence and exploration in rectifier networks. In: International Conference on Machine Learning. PMLR; 2017. p. 351\u201360."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_038","doi-asserted-by":"crossref","unstructured":"Casalicchio G, Molnar C, Bischl B. Visualizing the feature importance for black box models. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer; 2018. p. 655\u201370.","DOI":"10.1007\/978-3-030-10925-7_40"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_039","doi-asserted-by":"crossref","unstructured":"Lafta SA, Ismael MQ. Trip generation modeling for a selected sector in Baghdad city using the artificial neural network. J Intell Sys. 2022;31(1):356\u201369.","DOI":"10.1515\/jisys-2022-0023"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_040","doi-asserted-by":"crossref","unstructured":"Kim DW, Shin GY, Han MM. Analysis of feature importance and interpretation for malware classification. Comput Materials Continua. 2020;65(3):1891\u2013904.","DOI":"10.32604\/cmc.2020.010933"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_041","doi-asserted-by":"crossref","unstructured":"Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, et al. Explainable AI for trees: from local explanations to global understanding. 2019. arXiv: http:\/\/arXiv.org\/abs\/arXiv:190504610.","DOI":"10.1038\/s42256-019-0138-9"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_042","doi-asserted-by":"crossref","unstructured":"Aswad FM, Kareem AN, Khudhur AM, Khalaf BA, Mostafa SA. Tree-based machine learning algorithms in the Internet of Things environment for multivariate flood status prediction. J Intell Syst. 2022;31(1):1\u201314.","DOI":"10.1515\/jisys-2021-0179"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_043","doi-asserted-by":"crossref","unstructured":"Yekun EA, Haile AT. Student performance prediction with optimum multilabel ensemble model. J Intell Syst. 2021;30(1):511\u201323.","DOI":"10.1515\/jisys-2021-0016"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_044","unstructured":"Izza Y, Ignatiev A, Marques-Silva J. On explaining decision trees. 2020. arXiv: http:\/\/arXiv.org\/abs\/arXiv:201011034."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_045","doi-asserted-by":"crossref","unstructured":"Arik S\u00d6, Pfister T. TabNet: attentive interpretable tabular learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35; 2021. p. 6679\u201387.","DOI":"10.1609\/aaai.v35i8.16826"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_046","doi-asserted-by":"crossref","unstructured":"Ho A, Reich J, Nesterko S, Seaton D, Mullaney T, Waldo J, et al. HarvardX and MITx: the first year of open online courses, fall 2012-summer 2013. Ho, AD, Reich, J, Nesterko, S, Seaton, DT, Mullaney, T, Waldo, J, & Chuang, I(2014) HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No 1). 2014.","DOI":"10.2139\/ssrn.2381263"},{"key":"2025120517213984434_j_jisys-2022-0212_ref_047","unstructured":"HarvardX. HarvardX Person-Course Academic Year 2013 De-Identified dataset, version 3.0. 2014. 10.7910\/DVN\/26147."},{"key":"2025120517213984434_j_jisys-2022-0212_ref_048","doi-asserted-by":"crossref","unstructured":"Moro S, Cortez P, Rita P. A data-driven approach to predict the success of bank telemarketing. Decision Support Sys. 2014;62:22\u201331.","DOI":"10.1016\/j.dss.2014.03.001"}],"container-title":["Journal of Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/jisys-2022-0212\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/jisys-2022-0212\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T17:25:49Z","timestamp":1764955549000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/jisys-2022-0212\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,1]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1,12]]},"published-print":{"date-parts":[[2023,1,12]]}},"alternative-id":["10.1515\/jisys-2022-0212"],"URL":"https:\/\/doi.org\/10.1515\/jisys-2022-0212","relation":{},"ISSN":["2191-026X"],"issn-type":[{"type":"electronic","value":"2191-026X"}],"subject":[],"published":{"date-parts":[[2023,1,1]]},"article-number":"20220212"}}