{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:15:19Z","timestamp":1776111319426,"version":"3.50.1"},"reference-count":108,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T00:00:00Z","timestamp":1724976000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Research into explainable artificial intelligence (XAI) methods has exploded over the past five years. It is essential to synthesize and categorize this research and, for this purpose, multiple systematic reviews on XAI mapped out the landscape of the existing methods. To understand how these methods have developed and been applied and what evidence has been accumulated through model training and analysis, we carried out a tertiary literature review that takes as input systematic literature reviews published between 1992 and 2023. We evaluated 40 systematic literature review papers and presented binary tabular overviews of researched XAI methods and their respective characteristics, such as the scope, scale, input data, explanation data, and machine learning models researched. We identified seven distinct characteristics and organized them into twelve specific categories, culminating in the creation of comprehensive research grids. Within these research grids, we systematically documented the presence or absence of research mentions for each pairing of characteristic and category. We identified 14 combinations that are open to research. Our findings reveal a significant gap, particularly in categories like the cross-section of feature graphs and numerical data, which appear to be notably absent or insufficiently addressed in the existing body of research and thus represent a future research road map.<\/jats:p>","DOI":"10.3390\/make6030098","type":"journal-article","created":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T09:39:32Z","timestamp":1725010772000},"page":"1997-2017","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Tertiary Review on Explainable Artificial Intelligence: Where Do We Stand?"],"prefix":"10.3390","volume":"6","author":[{"given":"Frank","family":"van Mourik","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9322-0674","authenticated-orcid":false,"given":"Annemarie","family":"Jutte","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands"},{"name":"Ambient Intelligence Research Group, Saxion University of Applied Sciences, 7513 AB Enschede, The Netherlands"}]},{"given":"Stijn E.","family":"Berendse","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5978-2754","authenticated-orcid":false,"given":"Faiza A.","family":"Bukhsh","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2760-6892","authenticated-orcid":false,"given":"Faizan","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands"},{"name":"Ambient Intelligence Research Group, Saxion University of Applied Sciences, 7513 AB Enschede, The Netherlands"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in artificial intelligence: Insights from the social sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artif. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1007\/s11023-020-09529-4","article-title":"Algorithmic Fairness in Mortgage Lending: From Absolute Conditions to Relational Trade-offs","volume":"31","author":"Lee","year":"2021","journal-title":"Minds Mach."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Amann, J., Blasimme, A., Vayena, E., Frey, D., Madai, V.I., and consortium, t.P. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Med. Inform. Decis. Mak., 20.","DOI":"10.1186\/s12911-020-01332-6"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2176","DOI":"10.1038\/s41591-021-01595-0","article-title":"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations","volume":"27","author":"Zhang","year":"2021","journal-title":"Nat. Med."},{"key":"ref_5","unstructured":"European Commission (2021). Proposal for a Regulation of the European Parliament and of the Council on Artificial Intelligence and Amending Certain Union Legislative Acts. EUR-Lex, COM(2021) 206 final, Available online: https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX:52021PC0206."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Gianfagna, L., and Di Cecco, A. (2021). Explainable AI with Python, Springer.","DOI":"10.1007\/978-3-030-68640-6"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Islam, M., Ahmed, M., Barua, S., and Begum, S. (2022). A Systematic Review of Explainable Artificial Intelligence in Terms of Different Application Domains and Tasks. Appl. Sci., 12.","DOI":"10.3390\/app12031353"},{"key":"ref_8","unstructured":"Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, University of Durham. Technical Report EBSE-2007-01."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1136\/ebmental-2018-300014","article-title":"Ten simple rules for conducting umbrella reviews","volume":"21","author":"Radua","year":"2018","journal-title":"Evid. Based Ment. Health"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"110273","DOI":"10.1016\/j.knosys.2023.110273","article-title":"Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities","volume":"263","author":"Saeed","year":"2023","journal-title":"Knowl.-Based Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/s11192-015-1765-5","article-title":"The journal coverage of Web of Science and Scopus: A comparative analysis","volume":"106","author":"Mongeon","year":"2016","journal-title":"Scientometrics"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wohlin, C. (2014, January 13\u201314). Guidelines for snowballing in systematic literature studies and a replication in software engineering. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, London, UK.","DOI":"10.1145\/2601248.2601268"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"615","DOI":"10.3390\/make3030032","article-title":"Classification of Explainable Artificial Intelligence Methods through Their Output Formats","volume":"3","author":"Vilone","year":"2021","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"110592","DOI":"10.1016\/j.ejrad.2022.110592","article-title":"A systematic review on the use of explainability in deep learning systems for computer aided diagnosis in radiology: Limited use of explainable AI?","volume":"157","author":"Groen","year":"2022","journal-title":"Eur. J. Radiol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.neucom.2022.09.129","article-title":"Explaining deep neural networks: A survey on the global interpretation methods","volume":"513","author":"Saleem","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Pedreschi, D., and Giannotti, F. (2018). A Survey Of Methods For Explaining Black Box Models. arXiv.","DOI":"10.1145\/3236009"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1613\/jair.1.12228","article-title":"A Survey on the Explainability of Supervised Machine Learning","volume":"70","author":"Burkart","year":"2020","journal-title":"J. Artif. Intell. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1007\/s42979-021-00592-x","article-title":"Machine Learning: Algorithms, Real-World Applications and Research Directions","volume":"2","author":"Sarker","year":"2021","journal-title":"SN Comput. Sci."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3583558","article-title":"From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI","volume":"55","author":"Nauta","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_20","unstructured":"Wang, X., Wang, D., Xu, C., He, X., Cao, Y., and Chua, T.S. (February, January 27). Explainable reasoning over knowledge graphs for recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. arXiv.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_22","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"101209","DOI":"10.1016\/j.softx.2022.101209","article-title":"RuleXAI\u2014A package for rule-based explanations of machine learning model","volume":"20","author":"Macha","year":"2022","journal-title":"SoftwareX"},{"key":"ref_25","first-page":"24","article-title":"Extracting tree-structured representations of trained networks","volume":"8","author":"Craven","year":"1995","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_26","first-page":"1","article-title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets","volume":"29","author":"Chen","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","unstructured":"Molnar, C. (2020). Interpretable Machine Learning, Lulu.com."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.inffus.2021.11.003","article-title":"Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms, and applications","volume":"81","author":"Chou","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_29","first-page":"1","article-title":"Examples are not enough, learn to criticize! Criticism for interpretability","volume":"29","author":"Kim","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_30","unstructured":"Rajani, N.F., McCann, B., Xiong, C., and Socher, R. (August, January 28). Explain Yourself! Leveraging Language Models for Commonsense Reasoning. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"e7","DOI":"10.23915\/distill.00007","article-title":"Feature visualization","volume":"2","author":"Olah","year":"2017","journal-title":"Distill"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"103627","DOI":"10.1016\/j.artint.2021.103627","article-title":"Knowledge graphs as tools for explainable machine learning: A survey","volume":"302","author":"Tiddi","year":"2022","journal-title":"Artif. Intell."},{"key":"ref_33","first-page":"29","article-title":"A Survey of Data-Driven and Knowledge-Aware eXplainable AI","volume":"34","author":"Li","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5975","DOI":"10.1007\/s10462-022-10306-1","article-title":"Deep learning in drug discovery: An integrative review and future challenges","volume":"56","author":"Askr","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ahmed, S., Solis-Oba, R., and Ilie, L. (2022). Explainable-AI in Automated Medical Report Generation Using Chest X-ray Images. Appl. Sci., 12.","DOI":"10.3390\/app122211750"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Antoniadi, A., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B., and Mooney, C. (2021). Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: A systematic review. Appl. Sci., 11.","DOI":"10.3390\/app11115088"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Adak, A., Pradhan, B., and Shukla, N. (2022). Sentiment Analysis of Customer Reviews of Food Delivery Services Using Deep Learning and Explainable Artificial Intelligence: Systematic Review. Foods, 11.","DOI":"10.3390\/foods11101500"},{"key":"ref_38","unstructured":"Chakrobartty, S., and El-Gayar, O. (2021, January 9\u201313). Explainable artificial intelligence in the medical domain: A systematic review. Proceedings of the AMCIS 2021, Virtual."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Guidotti, R. (2022). Counterfactual explanations and how to find them: Literature review and benchmarking. Data Min. Knowl. Discov.","DOI":"10.1007\/s10618-022-00831-6"},{"key":"ref_40","first-page":"28","article-title":"A systematic review on model-agnostic XAI libraries","volume":"3017","author":"Darias","year":"2021","journal-title":"ICCBR Work."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Mechelli, A., and Vieira, S. (2020). Chapter 6\u2014Support vector machine. Machine Learning, Academic Press.","DOI":"10.1038\/s41537-020-0094-8"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ma\u00e7as, C., Campos, J.R., and Louren\u00e7o, N. (2023, January 25\u201328). Understanding the Forest: A Visualization Tool to Support Decision Tree Analysis. Proceedings of the 2023 27th International Conference Information Visualisation (IV), Tampere, Finland.","DOI":"10.1109\/IV60283.2023.00047"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1019","DOI":"10.1177\/01655515221112844","article-title":"Knowledge-graph-based explainable AI: A systematic review","volume":"50","author":"Rajabi","year":"2022","journal-title":"J. Inf. Sci."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.ejca.2022.02.025","article-title":"Explainable artificial intelligence in skin cancer recognition: A systematic review","volume":"167","author":"Hauser","year":"2022","journal-title":"Eur. J. Cancer"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Szymanski, M., Verbert, K., and Vanden Abeele, V. (2022, January 18\u201323). Designing and evaluating explainable AI for non-AI experts: Challenges and opportunities. Proceedings of the 16th ACM Conference on Recommender Systems, Seattle, WA, USA.","DOI":"10.1145\/3523227.3547427"},{"key":"ref_47","unstructured":"Das, A., and Rad, P. (2020). Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"962","DOI":"10.1109\/TITS.2022.3167957","article-title":"Lidar Point Cloud Compression, Processing and Learning for Autonomous Driving","volume":"24","author":"Abbasi","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"134355","DOI":"10.1109\/ACCESS.2023.3337049","article-title":"Point Cloud Analysis of Railway Infrastructure: A Systematic Literature Review","volume":"11","author":"Dekker","year":"2023","journal-title":"IEEE Access"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"27484","DOI":"10.1109\/ACCESS.2024.3366994","article-title":"Translating Image XAI to Multivariate Time Series","volume":"12","author":"Tronchin","year":"2024","journal-title":"IEEE Access"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"5834","DOI":"10.1109\/JSTARS.2024.3370159","article-title":"Explainable Artificial Intelligence for Machine Learning-Based Photogrammetric Point Cloud Classification","volume":"17","author":"Atik","year":"2024","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1744","DOI":"10.1109\/TMM.2019.2963592","article-title":"Pointhop: An explainable machine learning method for point cloud classification","volume":"22","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Multimed."},{"key":"ref_53","unstructured":"Ohsawa, Y. (2023). Interpretability and Explainability in Machine Learning. Living beyond Data: Toward Sustainable Value Creation, Springer International Publishing."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s11229-022-03485-5","article-title":"Conceptual challenges for interpretable machine learning","volume":"200","author":"Watson","year":"2022","journal-title":"Synthese"},{"key":"ref_55","first-page":"26598","article-title":"Shapley Residuals: Quantifying the limits of the Shapley value for explanations","volume":"34","author":"Kumar","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Nematzadeh, H., Garc\u00eda-Nieto, J., Navas-Delgado, I., and Aldana-Montes, J.F. (2023). Ensemble-based genetic algorithm explainer with automized image segmentation: A case study on melanoma detection dataset. Comput. Biol. Med., 155.","DOI":"10.1016\/j.compbiomed.2023.106613"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.infsof.2017.10.017","article-title":"Reporting systematic reviews: Some lessons from a tertiary study","volume":"95","author":"Budgen","year":"2018","journal-title":"Inf. Softw. Technol."},{"key":"ref_58","unstructured":"Holzinger, A., Saranti, A., Molnar, C., Biecek, P., and Samek, W. (2020, January 18). Explainable AI methods\u2014A brief overview. Proceedings of the International Workshop on Extending Explainable AI beyond Deep Models and Classifiers, Vienna, Austria."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"122120","DOI":"10.1016\/j.techfore.2022.122120","article-title":"Explainable Artificial Intelligence (XAI) from a user perspective: A synthesis of prior literature and problematizing avenues for future research","volume":"186","author":"Haque","year":"2023","journal-title":"Technol. Forecast. Soc. Chang."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1038\/s41746-022-00699-2","article-title":"Explainable medical imaging AI needs human-centered design: Guidelines and evidence from a systematic review","volume":"5","author":"Chen","year":"2022","journal-title":"Npj Digit. Med."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Loh, H., Ooi, C., Seoni, S., Barua, P., Molinari, F., and Acharya, U. (2022). Application of explainable artificial intelligence for healthcare: A systematic review of the last decade (2011\u20132022). Comput. Methods Programs Biomed., 226.","DOI":"10.1016\/j.cmpb.2022.107161"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Doh, R., Zhou, C., Arthur, J., Tawiah, I., and Doh, B. (2022). A Systematic Review of Deep Knowledge Graph-Based Recommender Systems, with Focus on Explainable Embeddings. Data, 7.","DOI":"10.3390\/data7070094"},{"key":"ref_63","unstructured":"Okolo, C., Dell, N., and Vashistha, A. (July, January 29). Making AI Explainable in the Global South: A Systematic Review. Proceedings of the 5th ACM SIGCAS\/SIGCHI Conference on Computing and Sustainable Societies, Seattle, WA, USA."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Yuan, J., and Bertini, E. (2022, January 12). Context Sight: Model Understanding and Debugging via Interpretable Context. Proceedings of the Workshop on Human-in-the-Loop Data Analytics, Philadelphia, PA, USA.","DOI":"10.1145\/3546930.3547502"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"108391","DOI":"10.1016\/j.asoc.2021.108391","article-title":"Interpretability in the medical field: A systematic mapping and review study","volume":"117","author":"Hakkoum","year":"2022","journal-title":"Appl. Soft Comput."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"107587","DOI":"10.1016\/j.knosys.2021.107587","article-title":"Explainability in supply chain operational risk management: A systematic literature review","volume":"235","author":"Nimmy","year":"2022","journal-title":"Knowl.-Based Syst."},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Engelmann, D., Damasio, J., Panisson, A., Mascardi, V., and Bordini, R. (2022, January 22\u201325). Argumentation as a Method for Explainable AI: A Systematic Literature Review. Proceedings of the 2022 17th Iberian Conference on Information Systems and Technologies (CISTI), Madrid, Spain.","DOI":"10.23919\/CISTI54924.2022.9820411"},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/RBME.2022.3185953","article-title":"Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Review","volume":"16","author":"Giuste","year":"2022","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"2112","DOI":"10.1016\/j.csbj.2022.04.021","article-title":"On the road to explainable AI in drug-drug interactions prediction: A systematic review","volume":"20","author":"Vo","year":"2022","journal-title":"Comput. Struct. Biotechnol. J."},{"key":"ref_70","doi-asserted-by":"crossref","unstructured":"Salahuddin, Z., Woodruff, H., Chatterjee, A., and Lambin, P. (2022). Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Comput. Biol. Med., 140.","DOI":"10.1016\/j.compbiomed.2021.105111"},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Nor, A., Pedapati, S., Muhammad, M., and Leiva, V. (2021). Overview of explainable artificial intelligence for prognostic and health management of industrial assets based on preferred reporting items for systematic reviews and meta-analyses. Sensors, 21.","DOI":"10.3390\/s21238020"},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.inffus.2021.05.009","article-title":"Notions of explainability and evaluation approaches for explainable artificial intelligence","volume":"76","author":"Vilone","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Wells, L., and Bednarz, T. (2021). Explainable AI and Reinforcement Learning\u2014A Systematic Review of Current Approaches and Trends. Front. Artif. Intell., 4.","DOI":"10.3389\/frai.2021.550030"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"153316","DOI":"10.1109\/ACCESS.2021.3127881","article-title":"A Systematic Review of Human-Computer Interaction and Explainable Artificial Intelligence in Healthcare with Artificial Intelligence Techniques","volume":"9","author":"Nazar","year":"2021","journal-title":"IEEE Access"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Chazette, L., Brunotte, W., and Speith, T. (2021, January 20\u201324). Exploring Explainability: A Definition, a Model, and a Knowledge Catalogue. Proceedings of the 2021 IEEE 29th International Requirements Engineering Conference (RE), Notre Dame, IN, USA.","DOI":"10.1109\/RE51729.2021.00025"},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"33132","DOI":"10.1109\/ACCESS.2021.3061368","article-title":"Explainable Student Performance Prediction Models: A Systematic Review","volume":"9","author":"Alamri","year":"2021","journal-title":"IEEE Access"},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"11974","DOI":"10.1109\/ACCESS.2021.3051315","article-title":"A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence","volume":"9","author":"Stepin","year":"2021","journal-title":"IEEE Access"},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"1173","DOI":"10.1093\/jamia\/ocaa053","article-title":"Explainable artificial intelligence models using real-world electronic health record data: A systematic scoping review","volume":"27","author":"Payrovnaziri","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1177\/1473871620904671","article-title":"A survey of surveys on the use of visualization for interpreting machine learning models","volume":"19","author":"Chatzimparmpas","year":"2020","journal-title":"Inf. Vis."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Grossberg, S. (2020). A Path Toward Explainable AI and Autonomous Adaptive Intelligence: Deep Learning, Adaptive Resonance, and Models of Perception, Emotion, and Action. Front. Neurorobot., 14.","DOI":"10.3389\/fnbot.2020.00036"},{"key":"ref_81","first-page":"10959","article-title":"Bibliometric analysis of passive image forgery detection and explainable AI","volume":"2020","author":"Kadam","year":"2020","journal-title":"Libr. Philos. Pract."},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/978-3-030-57321-8_18","article-title":"Scenario-Based Requirements Elicitation for User-Centric Explainable AI: A Case in Fraud Detection","volume":"Volume 12279 LNCS","author":"Cirqueira","year":"2020","journal-title":"Machine Learning and Knowledge Extraction"},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1007\/978-3-030-50316-1_13","article-title":"Explainable Recommendations in Intelligent Systems: Delivery Methods, Modalities and Risks","volume":"Volume 385 LNBIP","author":"Naiseh","year":"2020","journal-title":"Research Challenges in Information Science"},{"key":"ref_84","unstructured":"Chromik, M., and Schuessler, M. (2020, January 17). A taxonomy for human subject evaluation of black-box explanations in XAI. Proceedings of the ExSS-ATEC\u201920, Cagliari, Italy."},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Anjomshoae, S., Calvaresi, D., Najjar, A., and Framling, K. (2019, January 13\u201317). Explainable agents and robots: Results from a systematic literature review. Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, QC, Canada.","DOI":"10.1007\/978-3-030-30391-4"},{"key":"ref_86","first-page":"30","article-title":"Semantic web technologies for explainable machine learning models: A literature review","volume":"2465","author":"Seeliger","year":"2019","journal-title":"CEUR Workshop Proc."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"130733","DOI":"10.1109\/ACCESS.2022.3229233","article-title":"Safety Assurance of Artificial Intelligence-Based Systems: A Systematic Literature Review on the State of the Art and Guidelines for Future Work","volume":"10","author":"Neto","year":"2022","journal-title":"IEEE Access"},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"104298","DOI":"10.1016\/j.autcon.2022.104298","article-title":"Trustworthy AI and robotics: Implications for the AEC industry","volume":"139","author":"Emaminejad","year":"2022","journal-title":"Autom. Constr."},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.inffus.2022.11.013","article-title":"Beyond explaining: Opportunities and challenges of XAI-based model improvement","volume":"92","author":"Weber","year":"2023","journal-title":"Inf. Fusion"},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.dss.2018.01.002","article-title":"Disentangling consumer recommendations: Explaining and predicting airline recommendations based on online reviews","volume":"107","author":"Siering","year":"2018","journal-title":"Decis. Support Syst."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"107095","DOI":"10.1016\/j.chb.2021.107095","article-title":"Machine learning for suicidal ideation identification: A systematic literature review","volume":"128","author":"Heckler","year":"2022","journal-title":"Comput. Hum. Behav."},{"key":"ref_92","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1007\/s11257-017-9195-0","article-title":"A systematic review and taxonomy of explanations in decision support and recommender systems","volume":"27","author":"Nunes","year":"2017","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_93","unstructured":"Mary, S. (2019). Explainable Artificial Intelligence Applications in NLP, Biomedical, and Malware Classification: A Literature Review. Intelligent Computing, Springer."},{"key":"ref_94","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23.","DOI":"10.3390\/e23010018"},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Gulum, M.A., Trombley, C.M., and Kantardzic, M. (2021). A Review of Explainable Deep Learning Cancer Detection Models in Medical Imaging. Appl. Sci., 11.","DOI":"10.3390\/app11104573"},{"key":"ref_96","unstructured":"Verma, S., Boonsanong, V., Hoang, M., Hines, K.E., Dickerson, J.P., and Shah, C. (2020). Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review. arXiv."},{"key":"ref_97","unstructured":"Mueller, S.T., Hoffman, R.R., Clancey, W., Emrey, A., and Klein, G. (2019). Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI. arXiv."},{"key":"ref_98","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.inffus.2021.07.016","article-title":"Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond","volume":"77","author":"Yang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_99","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1016\/j.neucom.2010.02.016","article-title":"Rule extraction from support vector machines: A review","volume":"74","author":"Barakat","year":"2010","journal-title":"Neurocomputing"},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1108\/INTR-08-2021-0600","article-title":"How to explain AI systems to end users: A systematic literature review and research agenda","volume":"32","author":"Laato","year":"2022","journal-title":"Internet Res."},{"key":"ref_101","doi-asserted-by":"crossref","unstructured":"Gerlings, J., Shollo, A., and Constantiou, I. (2020). Reviewing the Need for Explainable Artificial Intelligence (xAI). arXiv.","DOI":"10.24251\/HICSS.2021.156"},{"key":"ref_102","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1017\/S026988890200019X","article-title":"A review of explanation methods for Bayesian networks","volume":"17","author":"Lacave","year":"2002","journal-title":"Knowl. Eng. Rev."},{"key":"ref_103","unstructured":"Vilone, G., and Longo, L. (2020). Explainable Artificial Intelligence: A Systematic Review. arXiv."},{"key":"ref_104","doi-asserted-by":"crossref","first-page":"155534342210851","DOI":"10.1177\/15553434221085114","article-title":"Examining Physicians\u2019 Explanatory Reasoning in Re-Diagnosis Scenarios for Improving AI Diagnostic Systems","volume":"16","author":"Alam","year":"2022","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/mp.15359","article-title":"A review of explainable and interpretable AI with applications in COVID-19 imaging","volume":"49","author":"Fuhrman","year":"2021","journal-title":"Med. Phys"},{"key":"ref_106","doi-asserted-by":"crossref","first-page":"59800","DOI":"10.1109\/ACCESS.2021.3070212","article-title":"A Review on Explainability in Multimodal Deep Neural Nets","volume":"9","author":"Joshi","year":"2021","journal-title":"IEEE Access"},{"key":"ref_107","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1111\/lnc3.12175","article-title":"Understanding Counterfactuality: A Review of Experimental Evidence for the Dual Meaning of Counterfactuals","volume":"10","author":"Kulakova","year":"2016","journal-title":"Lang Linguist Compass"},{"key":"ref_108","doi-asserted-by":"crossref","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: A comprehensive review","volume":"55","author":"Minh","year":"2022","journal-title":"Artif. Intell. Rev."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/3\/98\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:45:31Z","timestamp":1760111131000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/3\/98"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,30]]},"references-count":108,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["make6030098"],"URL":"https:\/\/doi.org\/10.3390\/make6030098","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,30]]}}}