{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T10:37:42Z","timestamp":1780483062188,"version":"3.54.1"},"reference-count":46,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T00:00:00Z","timestamp":1740009600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Commission Horizon Europe Program","award":["101057746"],"award-info":[{"award-number":["101057746"]}]},{"name":"European Commission Horizon Europe Program","award":["2200058"],"award-info":[{"award-number":["2200058"]}]},{"name":"Swiss State Secretariat for Education, Research and Innovation (SERI)","award":["101057746"],"award-info":[{"award-number":["101057746"]}]},{"name":"Swiss State Secretariat for Education, Research and Innovation (SERI)","award":["2200058"],"award-info":[{"award-number":["2200058"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Deep connectionist models are characterized by many neurons grouped together in many successive layers. As a result, their data classifications are difficult to understand. We present two novel algorithms which explain the responses of several black-box machine learning models. The first is Fidex, which is local and thus applied to a single sample. The second, called FidexGlo, is global and uses Fidex. Both algorithms generate explanations by means of propositional rules. In our framework, the discriminative boundaries are parallel to the input variables and their location is precisely determined. Fidex is a heuristic algorithm that, at each step, establishes where the best hyperplane is that has increased fidelity the most. The algorithmic complexity of Fidex is proportional to the maximum number of steps, the number of possible hyperplanes, which is finite, and the number of samples. We first used FidexGlo with ensembles and support vector machines (SVMs) to show that its performance on three benchmark problems is competitive in terms of complexity, fidelity and accuracy. The most challenging part was then to apply it to convolutional neural networks. We achieved this with three classification problems based on images. We obtained accurate results and described the characteristics of the rules generated, as well as several examples of explanations illustrated with their corresponding images. To the best of our knowledge, this is one of the few works showing a global rule extraction technique applied to both ensembles, SVMs and deep neural networks.<\/jats:p>","DOI":"10.3390\/a18030120","type":"journal-article","created":{"date-parts":[[2025,2,20]],"date-time":"2025-02-20T04:03:17Z","timestamp":1740024197000},"page":"120","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Fidex and FidexGlo: From Local Explanations to Global Explanations of Deep Models"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6070-3459","authenticated-orcid":false,"given":"Guido","family":"Bologna","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Applied Sciences and Arts of Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4138-204X","authenticated-orcid":false,"given":"Jean-Marc","family":"Boutay","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Applied Sciences and Arts of Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Damian","family":"Boquete","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Applied Sciences and Arts of Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Quentin","family":"Leblanc","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Applied Sciences and Arts of Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deniz","family":"K\u00f6pr\u00fcl\u00fc","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Applied Sciences and Arts of Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ludovic","family":"Pfeiffer","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Applied Sciences and Arts of Western Switzerland, Rue de la Prairie 4, 1202 Geneva, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,2,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3236009","article-title":"A survey of methods for explaining black box models","volume":"51","author":"Guidotti","year":"2018","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: A comprehensive review","volume":"55","author":"Minh","year":"2021","journal-title":"Artif. Intell. Rev."},{"key":"ref_4","unstructured":"Holzinger, A., Saranti, A., Molnar, C., Biecek, P., and Samek, W. (2020, January 18). Explainable AI methods a brief overview. Proceedings of the International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers, Vienna, Austria."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"105606","DOI":"10.1016\/j.engappai.2022.105606","article-title":"An analysis of explainability methods for convolutional neural networks","volume":"117","author":"Haar","year":"2023","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1016\/0950-7051(96)81920-4","article-title":"Survey and critique of techniques for extracting rules from trained artificial neural networks","volume":"8","author":"Andrews","year":"1995","journal-title":"Knowl.-Based Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Freund, Y., and Schapire, R.E. (1995, January 13\u201315). A desicion-theoretic generalization of on-line learning and an application to boosting. Proceedings of the European Conference on Computational Learning Theory, Barcelona, Spain.","DOI":"10.1007\/3-540-59119-2_166"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should i trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1631\/FITEE.1700808","article-title":"Visual interpretability for deep learning: A survey","volume":"19","author":"Zhang","year":"2018","journal-title":"Front. Inf. Technol. Electron. Eng."},{"key":"ref_11","first-page":"146","article-title":"Constraining the MLP power of expression to facilitate symbolic rule extraction","volume":"Volume 1","author":"Bologna","year":"1998","journal-title":"Proceedings of the 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No. 98CH36227)"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/j.jal.2004.03.004","article-title":"Is it worth generating rules from neural network ensembles?","volume":"2","author":"Bologna","year":"2004","journal-title":"J. Appl. Log."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bologna, G. (2021). A rule extraction technique applied to ensembles of neural networks, random forests, and gradient-boosted trees. Algorithms, 14.","DOI":"10.3390\/a14120339"},{"key":"ref_14","unstructured":"Bologna, G., Boutay, J.M., Leblanc, Q., and Boquete, D. (June, January 31). Fidex: An Algorithm for the Explainability of Ensembles and SVMs. Proceedings of the International Work-Conference on the Interplay Between Natural and Artificial Computation, Olh\u00e2o, Portugal."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Yang, Y., Wu, Y.N., and Zhu, S.C. (2018). Interpreting CNNs via decision trees. arXiv.","DOI":"10.1109\/CVPR.2019.00642"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Chen, H., Lundberg, S., and Lee, S.I. (2021). Explaining models by propagating Shapley values of local components. Explainable AI in Healthcare and Medicine: Building a Culture of Transparency and Accountability, Springer.","DOI":"10.1007\/978-3-030-53352-6_24"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1007\/BF00993309","article-title":"C4.5: Programs for machine learning. morgan kaufmann publishers, inc., 1993","volume":"16","author":"Quinlan","year":"1994","journal-title":"Mach. Learn."},{"key":"ref_21","unstructured":"Golea, M. (, January April). On the complexity of rule extraction from neural networks and network querying. Proceedings of the Rule Extraction from Trained Artificial Neural Networks Workshop, Society for the Study of Artificial Intelligence and Simulation of Behavior Workshop Series (AISB), Canberra, Australia."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Diederich, J. (2008). Rule Extraction from Support Vector Machines, Springer Science & Business Media.","DOI":"10.1007\/978-3-540-75390-2"},{"key":"ref_23","first-page":"24","article-title":"Extracting tree-structured representations of trained networks","volume":"8","author":"Craven","year":"1995","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/s11063-011-9207-8","article-title":"Reverse engineering the neural networks for rule extraction in classification problems","volume":"35","author":"Augasta","year":"2012","journal-title":"Neural Process. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Schaaf, N., Huber, M., and Maucher, J. (2019, January 16\u201319). Enhancing decision tree based interpretation of deep neural networks through l1-orthogonal regularization. Proceedings of the 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA), Boca Raton, FL, USA.","DOI":"10.1109\/ICMLA.2019.00016"},{"key":"ref_26","first-page":"3","article-title":"Extracting symbolic rules from trained neural network ensembles","volume":"16","author":"Zhou","year":"2003","journal-title":"Artif. Intell. Commun."},{"key":"ref_27","unstructured":"Johansson, U. (2007). Obtaining Accurate and Comprehensible Data Mining Models: An Evolutionary Approach, Link\u00f6ping University, Department of Computer and Information Science."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Hara, A., and Hayashi, Y. (2012, January 10\u201315). Ensemble neural network rule extraction using Re-RX algorithm. Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN), Brisbane, QLD, Australia.","DOI":"10.1109\/IJCNN.2012.6252446"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hayashi, Y., Sato, R., and Mitra, S. (2013, January 4\u20139). A new approach to three ensemble neural network rule extraction using recursive-rule extraction algorithm. Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN), Dallas, TX, USA.","DOI":"10.1109\/IJCNN.2013.6706823"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/j.procs.2019.09.182","article-title":"A new transparent ensemble method based on deep learning","volume":"159","author":"Sendi","year":"2019","journal-title":"Procedia Comput. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.cogsys.2022.07.004","article-title":"Rule extraction using ensemble of neural network ensembles","volume":"75","author":"Chakraborty","year":"2022","journal-title":"Cogn. Syst. Res."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Friedman, J.H., and Popescu, B.E. (2008). Predictive learning via rule ensembles. The Annals of Applied Statistics, Institute of Mathematical Statistics.","DOI":"10.1214\/07-AOAS148"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1707","DOI":"10.1142\/S0219622017500055","article-title":"Rule Extraction from Decision Trees Ensembles: New Algorithms Based on Heuristic Search and Sparse Group Lasso Methods","volume":"16","author":"Mashayekhi","year":"2017","journal-title":"Int. J. Inf. Technol. Decis. Mak."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Arrieta","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.eswa.2019.02.012","article-title":"RuleCOSI: Combination and simplification of production rules from boosted decision trees for imbalanced classification","volume":"126","author":"Obregon","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bologna, G., and Hayashi, Y. (2015, January 10\u201312). QSVM: A Support Vector Machine for rule extraction. Proceedings of the International Work-Conference on Artificial Neural Networks, Palma de Mallorca, Spain.","DOI":"10.1007\/978-3-319-19222-2_23"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bologna, G., and Hayashi, Y. (2018). A rule extraction study from svm on sentiment analysis. Big Data Cogn. Comput., 2.","DOI":"10.3390\/bdcc2010006"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"988","DOI":"10.1109\/72.788640","article-title":"An overview of statistical learning theory","volume":"10","author":"Vapnik","year":"1999","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_40","unstructured":"Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer Science & Business Media."},{"key":"ref_41","unstructured":"Lichman, M. (2013). UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences."},{"key":"ref_42","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1007\/s00365-006-0663-2","article-title":"On early stopping in gradient descent learning","volume":"26","author":"Yao","year":"2007","journal-title":"Constr. Approx."},{"key":"ref_44","unstructured":"LeCun, Y. (2021, April 06). LeNet-5, Convolutional Neural Networks. 2015. p. 20. Available online: http:\/\/yann.lecun.com\/exdb\/lenet."},{"key":"ref_45","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"\u00d6zgenel, \u00c7.F., and Sorgu\u00e7, A.G. (2018, January 20\u201325). Performance comparison of pretrained convolutional neural networks on crack detection in buildings. Proceedings of the ISARC, International Symposium on Automation and Robotics in Construction, Berlin, Germany.","DOI":"10.22260\/ISARC2018\/0094"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/3\/120\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:38:33Z","timestamp":1760027913000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/18\/3\/120"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,20]]},"references-count":46,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["a18030120"],"URL":"https:\/\/doi.org\/10.3390\/a18030120","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,20]]}}}