{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T03:24:11Z","timestamp":1784690651618,"version":"3.55.0"},"reference-count":50,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T00:00:00Z","timestamp":1753315200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>Artificial intelligence (AI) has emerged as a transformative force across industries, driven by advances in deep learning and natural language processing, and fueled by large-scale data and computing resources. Despite its rapid adoption, the opacity of AI systems poses significant challenges to trust and acceptance. This work explores the intersection of connectionist and symbolic approaches to artificial intelligence, focusing on the derivation of interpretable symbolic models, such as decision trees, from feedforward neural networks (FNNs). Decision trees provide a transparent framework for elucidating the operations of neural networks while preserving their functionality. The derivation is presented in a step-by-step approach and illustrated with several examples. A systematic methodology is proposed to bridge neural and symbolic paradigms by exploiting distributed representations in FNNs to identify symbolic components, including fillers, roles, and their interrelationships. The process traces neuron activation values and input configurations across network layers, mapping activations and their underlying inputs to decision tree edges. The resulting symbolic structures effectively capture FNN decision processes and enable scalability to deeper networks through iterative refinement of subpaths for each hidden layer. To validate the theoretical framework, a prototype was developed using Keras <jats:monospace>.h5<\/jats:monospace>-data and emulating TensorFlow within the Java JDK\/JavaFX environment. This prototype demonstrates the feasibility of extracting symbolic representations from neural networks, enhancing trust in AI systems, and promoting accountability.<\/jats:p>","DOI":"10.3389\/frai.2025.1618149","type":"journal-article","created":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T12:58:46Z","timestamp":1753361926000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Deriving equivalent symbol-based decision models from feedforward neural networks"],"prefix":"10.3389","volume":"8","author":[{"given":"Sebastian","family":"Seidel","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Uwe M.","family":"Borghoff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,7,24]]},"reference":[{"key":"B1","unstructured":"Abadi\n              M.\n            \n            \n              Agarwal\n              A.\n            \n            \n              Barham\n              P.\n            \n            \n              Brevdo\n              E.\n            \n            \n              Chen\n              Z.\n            \n            \n              Citro\n              C.\n            \n          \n          TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems\n          \n          2015"},{"key":"B2","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"B3","doi-asserted-by":"publisher","first-page":"19823","DOI":"10.1109\/JIOT.2024.3369852","article-title":"A novel internet of things web attack detection architecture based on the combination of symbolism and connectionism AI","volume":"11","author":"An","year":"2024","journal-title":"IEEE Internet Things J"},{"key":"B4","doi-asserted-by":"publisher","first-page":"e0130140","DOI":"10.1371\/journal.pone.0130140","article-title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","volume":"10","author":"Bach","year":"2015","journal-title":"PLoS ONE"},{"key":"B5","doi-asserted-by":"publisher","first-page":"2420","DOI":"10.1016\/j.neucom.2007.10.028","article-title":"Connectionist model generation: a first-order approach","volume":"71","author":"Bader","year":"2008","journal-title":"Neurocomputing"},{"key":"B6","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1109\/45.329294","article-title":"Feed-forward neural networks","volume":"13","author":"Bebis","year":"1994","journal-title":"IEEE Potentials"},{"key":"B7","doi-asserted-by":"publisher","first-page":"77110","DOI":"10.1109\/ACCESS.2022.3191790","article-title":"Artificial intelligence crime: An overview of malicious use and abuse of AI","volume":"10","author":"Blauth","year":"2022","journal-title":"IEEE Access"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1579166","DOI":"10.3389\/fhumd.2025.1579166","article-title":"Human-artificial interaction in the age of agentic AI: a system-theoretical approach","volume":"7","author":"Borghoff","year":"","journal-title":"Front. Human Dynam"},{"key":"B9","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1007\/s10791-025-09667-2","article-title":"An organizational theory for multi-agent interactions integrating human agents, LLMs, and specialized AI","volume":"28","author":"Borghoff","year":"","journal-title":"Discov. Computing"},{"key":"B10","doi-asserted-by":"publisher","first-page":"1331","DOI":"10.1016\/S0893-6080(01)00109-5","article-title":"Connectionist inference models","volume":"14","author":"Browne","year":"2001","journal-title":"Neural Netw"},{"key":"B11","doi-asserted-by":"publisher","first-page":"78994","DOI":"10.1109\/ACCESS.2023.3294569","article-title":"A review of trustworthy and explainable artificial intelligence (XAI)","volume":"11","author":"Chamola","year":"2023","journal-title":"IEEE Access"},{"key":"B12","unstructured":"Chollet\n              F.\n            \n          \n          Keras\n          \n          2015"},{"key":"B13","doi-asserted-by":"publisher","first-page":"78","DOI":"10.3390\/make5010006","article-title":"XAIR: A systematic metareview of explainable AI (XAI) aligned to the software development process","volume":"5","author":"Clement","year":"2023","journal-title":"Mach. Learn. Knowl. Extract"},{"key":"B14","doi-asserted-by":"publisher","first-page":"4765","DOI":"10.1007\/s10462-022-10275-5","article-title":"Recent advances in decision trees: an updated survey","volume":"56","author":"Costa","year":"2023","journal-title":"Artif. Intell. Rev"},{"key":"B15","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1109\/CBS.2018.8612259","article-title":"\u201cUnsupervised learning based on artificial neural network: a review,\u201d","volume-title":"2018 IEEE International Conference on Cyborg and Bionic Systems (CBS)","author":"Dike","year":"2018"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3561048","article-title":"Explainable AI (XAI): core ideas, techniques, and solutions","volume":"194","author":"Dwivedi","year":"2023","journal-title":"ACM Comput. Surv"},{"key":"B17","doi-asserted-by":"crossref","first-page":"826","DOI":"10.1109\/TSMC.1983.6313076","article-title":"\u201cNeocognitron: a neural network model for a mechanism of visual pattern recognition,\u201d","author":"Fukushima","year":"1983","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics, SMC-13"},{"key":"B18","volume-title":"Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow","author":"G\u00e9ron","year":"2022"},{"key":"B19","first-page":"80","article-title":"\u201cExplaining explanations: An overview of interpretability of machine learning,\u201d","volume-title":"5th IEEE International Conference on Data Science and Advanced Analytics, DSAA 2018, Turin, Italy","author":"Gilpin","year":"2018"},{"key":"B20","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"key":"B21","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1109\/MC.2018.3620965","article-title":"Toward human-understandable, explainable AI","volume":"51","author":"Hagras","year":"2018","journal-title":"Computer"},{"key":"B22","unstructured":"Introduction to HDF5\n          \n          2006"},{"key":"B23","doi-asserted-by":"publisher","first-page":"sp006837","DOI":"10.1113\/jphysiol.1962.sp006837","article-title":"Receptive fields, binocular interaction and functional architecture in the cat's visual cortex","volume":"160","author":"Hubel","year":"1962","journal-title":"J. Physiol"},{"key":"B24","unstructured":"Einf\u00fchrung in JSON\n          \n          2017"},{"key":"B25","first-page":"4765","article-title":"\u201cA unified approach to interpreting model predictions,\u201d","author":"Lundberg","year":"2017","journal-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017"},{"key":"B26","author":"Millington","year":"2009","journal-title":"Artificial Intelligence for Games"},{"key":"B27","unstructured":"\u201cLogical vs. analogical or symbolic vs. connectionist or neat vs. scruffy,\u201d\n          \n          \n            \n              Minsky\n              M.\n            \n          \n          Artificial Intelligence at MIT: Expanding Frontiers\n          \n          1991"},{"key":"B28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","article-title":"Methods for interpreting and understanding deep neural networks","volume":"73","author":"Montavon","year":"2018","journal-title":"Digit. Signal Process"},{"key":"B29","unstructured":"Mudd\n              J.\n            \n          \n          jHDF\n          \n          2015"},{"key":"B30","unstructured":"National Institute of Standards and Technology: AI Risk Management Framework\n          \n          2023"},{"key":"B31","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1007\/978-3-031-82949-9_2","article-title":"\u201cThe application of named entity recognition in military intelligence,\u201d","author":"Nitzl","year":""},{"key":"B32","doi-asserted-by":"publisher","first-page":"1540450","DOI":"10.3389\/fhumd.2025.1540450","article-title":"The use of artificial intelligence in military intelligence: An experimental investigation of added value in the analysis process","volume":"7","author":"Nitzl","year":"","journal-title":"Front. Human Dynam"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2501.14322","article-title":"Relative layer-wise relevance propagation: a more robust neural networks explaination","author":"Nyiri","year":"2025","journal-title":"arXiv"},{"key":"B34","unstructured":"\u201cFour principles of explainable artificial intelligence,\u201d\n          \n          \n            \n              Phillips\n              P. J.\n            \n            \n              Hahn\n              C. A.\n            \n            \n              Fontana\n              P. C.\n            \n            \n              Yates\n              A. N.\n            \n            \n              Greene\n              K.\n            \n            \n              Broniatowski\n              D. A.\n            \n            \n              Przybocki\n              M. A.\n            \n          \n          NIST Interagency\/Internal Report (NISTIR)\n          \n          2021"},{"key":"B35","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1109\/TAI.2021.3133846","article-title":"Recent advances in trustworthy explainable artificial intelligence: Status, challenges, and perspectives","volume":"3","author":"Rawal","year":"2022","journal-title":"IEEE Trans. Artif. Intell"},{"key":"B36","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1007\/0-387-25465-X_9","article-title":"\u201cDecision trees,\u201d","volume-title":"The Data Mining and Knowledge Discovery Handbook","author":"Rokach","year":"2005"},{"key":"B37","doi-asserted-by":"publisher","first-page":"386","DOI":"10.1037\/h0042519","article-title":"The perceptron: a probabilistic model for information storage and organization in the brain","volume":"65","author":"Rosenblatt","year":"1958","journal-title":"Psychol. Rev"},{"key":"B38","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/5236.001.0001","volume-title":"Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations","author":"Rumelhart","year":"1986"},{"key":"B39","volume-title":"Artificial Intelligence-A Modern Approach","author":"Russel","year":"2010"},{"key":"B40","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s43681-023-00406-1","article-title":"On the evaluation of the symbolic knowledge extracted from black boxes","volume":"4","author":"Sabbatini","year":"2024","journal-title":"AI Ethics"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1404.7828","article-title":"Deep learning in neural networks: an overview","author":"Schmidhuber","year":"2014","journal-title":"arXiv"},{"key":"B42","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1109\/CogInfoCom.2018.8639897","article-title":"\u201cExtracting state machines from feedforward neural networks,\u201d","volume-title":"2018 9th IEEE International Conference on Cognitive Infocommunications (CogInfoCom)","author":"Seidel","year":"2018"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2504.12446","article-title":"\u201cDeriving equivalent symbol-based decision models from feedforward neural networks,\u201d","author":"Seidel","year":"2025","journal-title":"arXiv"},{"key":"B44","first-page":"616","article-title":"\u201cUnderstanding neural network decisions by creating equivalent symbolic AI models,\u201d","volume-title":"Intelligent Systems and Applications. IntelliSys 2018. Advances in Intelligent Systems and Computing, vol. 868","author":"Seidel","year":"2019"},{"key":"B45","volume-title":"The Harmonic Mind: From Neural Computation to Optimality-Theoretic Grammar (Cognitive architecture)","author":"Smolensky","year":"2006"},{"key":"B46","first-page":"3319","article-title":"\u201cAxiomatic attribution for deep networks,\u201d","author":"Sundararajan","year":"2017","journal-title":"Proceedings of the 34th International Conference on Machine Learning, ICML 2017"},{"key":"B47","unstructured":"\u201cExplainable Artificial Intelligence,\u201d\n          \n          \n            \n              Turek\n              M.\n            \n          \n          The U.S. Defense Advanced Research Projects Agency\n          \n          2023"},{"key":"B48","unstructured":"United States Government: Executive Order on AI\n          \n          2025"},{"key":"B49","unstructured":"van Veen\n              F.\n            \n          \n          The Neural Network Zoo\n          \n          2016"},{"key":"B50","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1109\/TETCI.2021.3100641","article-title":"A survey on neural network interpretability","volume":"5","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1618149\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T12:58:52Z","timestamp":1753361932000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1618149\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,24]]},"references-count":50,"alternative-id":["10.3389\/frai.2025.1618149"],"URL":"https:\/\/doi.org\/10.3389\/frai.2025.1618149","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,24]]},"article-number":"1618149"}}