{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T03:38:34Z","timestamp":1769830714100,"version":"3.49.0"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T00:00:00Z","timestamp":1767225600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T00:00:00Z","timestamp":1767830400000},"content-version":"vor","delay-in-days":7,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100011914","name":"M\u00e4lardalen University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100011914","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2026,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Rectified linear unit (ReLU) based neural networks (NNs) are recognised for their remarkable accuracy. However, the decision-making processes of these networks are often complex and difficult to understand. This complexity can lead to challenges in error identification, establishing trust, and conducting thorough analyses. Existing methods often fail to provide clear insights into the actual computations occurring within each layer of these networks. To address this challenge, this study introduces a mechanistic interpretability method called ReLU Region Reasoning (Re3). This method uses the known piecewise-linear characteristics of ReLU networks to offer insights into neuron activation and accurately assess how each feature contributes to the final output and probability. Re3 effectively determines neuron activations and evaluates the contribution of each feature within a specified linear region. Experiments conducted on multiple benchmark datasets, including both tabular and image data, demonstrate that Re3 can replicate individual predictions without error, align feature importance with domain expertise, and maintain consistency with current explanatory methods, thereby avoiding the typical randomness. Analysing neurons reveals activation sparsity and identifies dominant units, thus providing clear targets for model simplification and troubleshooting. By ensuring transparency and algebraic accessibility in each stage of a ReLU-based NN\u2019s decision process, Re3 can be a valuable practical tool for achieving precise mechanistic interpretability.<\/jats:p>","DOI":"10.1007\/s10994-025-06957-0","type":"journal-article","created":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T22:40:54Z","timestamp":1767912054000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Mechanistic Interpretability of ReLU Neural Networks Through Piecewise-Affine Mapping"],"prefix":"10.1007","volume":"115","author":[{"given":"Arnab","family":"Barua","sequence":"first","affiliation":[]},{"given":"Mobyen Uddin","family":"Ahmed","sequence":"additional","affiliation":[]},{"given":"Shahina","family":"Begum","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2026,1,8]]},"reference":[{"key":"6957_CR1","unstructured":"Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In Advances in neural information processing systems (Vol. 31). ACM."},{"issue":"5","key":"6957_CR2","doi-asserted-by":"publisher","first-page":"717","DOI":"10.1016\/S0731-7085(99)00272-1","volume":"22","author":"S Agatonovic-Kustrin","year":"2000","unstructured":"Agatonovic-Kustrin, S., & Beresford, R. (2000). Basic concepts of artificial neural network (ANN) modeling and its application in pharmaceutical research. Journal of Pharmaceutical and Biomedical Analysis, 22(5), 717\u2013727.","journal-title":"Journal of Pharmaceutical and Biomedical Analysis"},{"key":"6957_CR3","doi-asserted-by":"crossref","unstructured":"Aggarwal, C. C., et\u00a0al. (2018). Neural networks and deep learning (Vol. 10). Springer.","DOI":"10.1007\/978-3-319-94463-0"},{"issue":"7","key":"6957_CR4","doi-asserted-by":"publisher","first-page":"0130140","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.-R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 0130140.","journal-title":"PLoS ONE"},{"key":"6957_CR5","doi-asserted-by":"crossref","unstructured":"Bak, S., Tran, H.-D., Hobbs, K., & Johnson, T. T. (2020). Improved geometric path enumeration for verifying relu neural networks. In International conference on computer aided verification (pp. 66\u201396). Springer.","DOI":"10.1007\/978-3-030-53288-8_4"},{"key":"6957_CR6","unstructured":"Berzins, A. (2023). Polyhedral complex extraction from relu networks using edge subdivision. In International conference on machine learning (pp. 2234\u20132244). PMLR."},{"key":"6957_CR7","unstructured":"Christoph, M. (2020). Interpretable machine learning: A guide for making black box models explainable. SAGE."},{"key":"6957_CR8","doi-asserted-by":"crossref","unstructured":"Chu, L., Hu, X., Hu, J., Wang, L., & Pei, J. (2018). Exact and consistent interpretation for piecewise linear neural networks: A closed form solution. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1244\u20131253).","DOI":"10.1145\/3219819.3220063"},{"issue":"209","key":"6957_CR9","first-page":"1","volume":"22","author":"I Covert","year":"2021","unstructured":"Covert, I., Lundberg, S., & Lee, S.-I. (2021). Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209), 1\u201390.","journal-title":"Journal of Machine Learning Research"},{"key":"6957_CR10","unstructured":"Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint. arXiv:1702.08608"},{"key":"6957_CR11","unstructured":"Garreau, D., & Luxburg, U. (2020). Explaining the explainer: A first theoretical analysis of lime. In International conference on artificial intelligence and statistics (pp. 1287\u20131296). PMLR."},{"key":"6957_CR12","unstructured":"Goodfellow, I., Bengio, Y., Courville, A.,& Bengio, Y. (2016). Deep learning (Vol. 1). MIT."},{"issue":"5","key":"6957_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM Computing Surveys (CSUR), 51(5), 1\u201342.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"6957_CR14","unstructured":"Hooker, S., Erhan, D., Kindermans, P.-J., & Kim, B. (2019). A benchmark for interpretability methods in deep neural networks. In Advances in neural information processing systems (Vol. 32)."},{"issue":"3","key":"6957_CR15","doi-asserted-by":"publisher","first-page":"1353","DOI":"10.3390\/app12031353","volume":"12","author":"MR Islam","year":"2022","unstructured":"Islam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022). A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Applied Sciences, 12(3), 1353.","journal-title":"Applied Sciences"},{"key":"6957_CR16","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (Vol. 25, pp. 1097\u20131105)."},{"key":"6957_CR17","unstructured":"Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020). Problems with shapley-value-based explanations as feature importance measures. In International conference on machine learning (pp. 5491\u20135500). PMLR."},{"key":"6957_CR18","unstructured":"Lederer, J. (2021). Activation functions in artificial neural networks: A systematic overview. arXiv preprint. arXiv:2101.09957"},{"issue":"3","key":"6957_CR19","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton, Z. C. (2018). The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3), 31\u201357.","journal-title":"Queue"},{"key":"6957_CR20","doi-asserted-by":"crossref","unstructured":"Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision (pp. 2736\u20132744).","DOI":"10.1109\/ICCV.2017.298"},{"key":"6957_CR21","unstructured":"Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in neural information processing systems (Vol. 30)."},{"issue":"8","key":"6957_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3546577","volume":"55","author":"A Madsen","year":"2022","unstructured":"Madsen, A., Reddy, S., & Chandar, S. (2022). Post-hoc interpretability for neural NLP: A survey. ACM Computing Surveys, 55(8), 1\u201342.","journal-title":"ACM Computing Surveys"},{"key":"6957_CR23","unstructured":"Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint. arXiv:1611.06440"},{"key":"6957_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","volume":"73","author":"G Montavon","year":"2018","unstructured":"Montavon, G., Samek, W., & M\u00fcller, K.-R. (2018). Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73, 1\u201315.","journal-title":"Digital Signal Processing"},{"key":"6957_CR25","unstructured":"Montufar, G. F., Pascanu, R., Cho, K., & Bengio, Y. (2014). On the number of linear regions of deep neural networks. In Advances in neural information processing systems (Vol. 27)."},{"issue":"44","key":"6957_CR26","doi-asserted-by":"publisher","first-page":"22071","DOI":"10.1073\/pnas.1900654116","volume":"116","author":"WJ Murdoch","year":"2019","unstructured":"Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019). Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences of the United States of America, 116(44), 22071\u201322080.","journal-title":"Proceedings of the National Academy of Sciences of the United States of America"},{"key":"6957_CR27","unstructured":"Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (pp. 807\u2013814)."},{"key":"6957_CR28","unstructured":"Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint. arXiv:2301.05217"},{"issue":"3","key":"6957_CR29","doi-asserted-by":"publisher","first-page":"10","DOI":"10.23915\/distill.00010","volume":"3","author":"C Olah","year":"2018","unstructured":"Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018). The building blocks of interpretability. Distill, 3(3), 10.","journal-title":"Distill"},{"key":"6957_CR30","unstructured":"Pascanu, R., Montufar, G., & Bengio, Y. (2013). On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv preprint. arXiv:1312.6098"},{"key":"6957_CR31","unstructured":"Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., & Sohl-Dickstein, J. (2017). On the expressive power of deep neural networks. In International conference on machine learning (pp. 2847\u20132854). PMLR."},{"key":"6957_CR32","doi-asserted-by":"crossref","unstructured":"Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). why should I trust you? explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135\u20131144).","DOI":"10.1145\/2939672.2939778"},{"issue":"6088","key":"6957_CR34","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1038\/323533a0","volume":"323","author":"DE Rumelhart","year":"1986","unstructured":"Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533\u2013536.","journal-title":"Nature"},{"issue":"1","key":"6957_CR35","doi-asserted-by":"publisher","first-page":"2400304","DOI":"10.1002\/aisy.202400304","volume":"7","author":"AM Salih","year":"2025","unstructured":"Salih, A. M., Raisi-Estabragh, Z., Galazzo, I. B., Radeva, P., Petersen, S. E., Lekadir, K., & Menegaz, G. (2025). A perspective on explainable artificial intelligence methods: Shap and lime. Advanced Intelligent Systems, 7(1), 2400304.","journal-title":"Advanced Intelligent Systems"},{"issue":"3","key":"6957_CR36","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1109\/JPROC.2021.3060483","volume":"109","author":"W Samek","year":"2021","unstructured":"Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & M\u00fcller, K.-R. (2021). Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3), 247\u2013278.","journal-title":"Proceedings of the IEEE"},{"issue":"1","key":"6957_CR37","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1007\/s44244-023-00009-z","volume":"1","author":"A Stadlhofer","year":"2023","unstructured":"Stadlhofer, A., & Mezhuyev, V. (2023). Approach to provide interpretability in machine learning models for image classification. Industrial Artificial Intelligence, 1(1), 10.","journal-title":"Industrial artificial intelligence"},{"key":"6957_CR38","unstructured":"Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning (pp. 3319\u20133328). PMLR."},{"key":"6957_CR39","unstructured":"Trimmel, M., Petzka, H., & Sminchisescu, C. (2021). Tropex: An algorithm for extracting linear terms in deep neural networks. In International conference on learning representations."},{"key":"6957_CR40","doi-asserted-by":"crossref","unstructured":"Velmurugan, M., Ouyang, C., Sindhgatta, R., & Moreira, C. (2023). Through the looking glass: Evaluating post hoc explanations using transparent models. International Journal of Data Science and Analytics, 20(2), 615\u2013635.","DOI":"10.1007\/s41060-023-00445-1"},{"issue":"2","key":"6957_CR41","first-page":"841","volume":"31","author":"S Wachter","year":"2017","unstructured":"Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harvard Journal of Law & Technology, 31(2), 841\u2013887.","journal-title":"Harvard Journal of Law & Technology"},{"key":"6957_CR42","doi-asserted-by":"publisher","DOI":"10.1016\/j.adhoc.2020.102346","volume":"111","author":"L Wang","year":"2021","unstructured":"Wang, L., Wang, C., Li, Y., & Wang, R. (2021). Explaining the behavior of neuron activations in deep neural networks. Ad Hoc Networks, 111, Article 102346.","journal-title":"Ad Hoc Networks"},{"key":"6957_CR43","doi-asserted-by":"crossref","unstructured":"Yu, R., Li, A., Chen, C.-F., Lai, J.-H., Morariu, V.I., Han, X., Gao, M., Lin, C.-Y., & Davis, L. S. (2018). NISP: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 9194\u20139203).","DOI":"10.1109\/CVPR.2018.00958"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06957-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-025-06957-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-025-06957-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T14:05:25Z","timestamp":1769781925000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-025-06957-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1]]},"references-count":42,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1]]}},"alternative-id":["6957"],"URL":"https:\/\/doi.org\/10.1007\/s10994-025-06957-0","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1]]},"assertion":[{"value":"29 July 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 December 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 December 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 January 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}}],"article-number":"17"}}