{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T05:15:31Z","timestamp":1781673331642,"version":"3.54.5"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T00:00:00Z","timestamp":1649808000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T00:00:00Z","timestamp":1649808000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007512","name":"Universitat Rovira i Virgili","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007512","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Artificial intelligence (AI) is used for various purposes that are critical to human life. However, most state-of-the-art AI algorithms are black-box models, which means that humans cannot understand how such models make decisions. To forestall an algorithm-based authoritarian society, decisions based on machine learning ought to inspire trust by being <jats:italic>explainable<\/jats:italic>. For AI explainability to be practical, it must be feasible to obtain explanations systematically and automatically. A usual methodology to explain predictions made by a (black-box) deep learning model is to build a surrogate model based on a less difficult, more understandable decision algorithm. In this work, we focus on explaining by means of model surrogates the (mis)behavior of black-box models trained via federated learning. Federated learning is a decentralized machine learning technique that aggregates partial models trained by a set of peers on their own private data to obtain a global model. Due to its decentralized nature, federated learning offers some privacy protection to the participating peers. Nonetheless, it remains vulnerable to a variety of security attacks and even to sophisticated privacy attacks. To mitigate the effects of such attacks, we turn to the causes underlying misclassification by the federated model, which may indicate manipulations of the model. Our approach is to use random forests containing decision trees of restricted depth as surrogates of the federated black-box model. Then, we leverage decision trees in the forest to compute the importance of the features involved in the wrong predictions. We have applied our method to detect security and privacy attacks that malicious peers or the model manager may orchestrate in federated learning scenarios. Empirical results show that our method can detect attacks with high accuracy and, unlike other attack detection mechanisms, it can also <jats:italic>explain<\/jats:italic> the operation of such attacks at the peers\u2019 side.<\/jats:p>","DOI":"10.1007\/s10489-022-03435-1","type":"journal-article","created":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T15:03:23Z","timestamp":1649862203000},"page":"169-185","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Explaining predictions and attacks in federated learning via random forests"],"prefix":"10.1007","volume":"53","author":[{"given":"Rami","family":"Haffar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7275-7887","authenticated-orcid":false,"given":"David","family":"S\u00e1nchez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josep","family":"Domingo-Ferrer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,4,13]]},"reference":[{"issue":"3\u20134","key":"3435_CR1","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1561\/2000000039","volume":"7","author":"L Deng","year":"2014","unstructured":"Deng L, Yu D (2014) Deep learning: methods and applications. Found Trends Signal Process 7(3\u20134):197\u2013387","journal-title":"Found Trends Signal Process"},{"issue":"7553","key":"3435_CR2","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436\u2013444","journal-title":"Nature"},{"key":"3435_CR3","doi-asserted-by":"publisher","first-page":"62","DOI":"10.3389\/fams.2018.00062","volume":"4","author":"J Kone\u010dny\u0300","year":"2018","unstructured":"Kone\u010dny\u0300 J, Richt\u00e1rik P (2018) Randomized distributed mean estimation: accuracy vs. communication. Front Appl Math Stat 4:62","journal-title":"Front Appl Math Stat"},{"key":"3435_CR4","unstructured":"Kone\u010dny\u0300 J, McMahan H B, Felix X Y, Richt\u00e1rik P, Suresh A T, Bacon D (2016) Federated learning: Strategies for improving communication efficiency"},{"issue":"1-88","key":"3435_CR5","first-page":"294","volume":"59","author":"GDP Regulation","year":"2016","unstructured":"Regulation G D P (2016) Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46. Official J Eur Union (OJ) 59(1-88):294","journal-title":"Official J Eur Union (OJ)"},{"key":"3435_CR6","unstructured":"European Commission\u2019s High-Level Expert Group on Artificial Intelligence: Ethics Guidelines for Trustworthy AI (2019). https:\/\/ec.europa.eu\/futurium\/en\/ai-alliance-consultation"},{"key":"3435_CR7","doi-asserted-by":"crossref","unstructured":"Shahriari K, Shahriari M (2017) IEEE standard review. Ethically aligned design: a vision for prioritizing human wellbeing with artificial intelligence and autonomous systems. In: 2017 IEEE Canada International Humanitarian Technology Conference (IHTC). IEEE, pp 197\u2013201","DOI":"10.1109\/IHTC.2017.8058187"},{"key":"3435_CR8","unstructured":"(2021). Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts. https:\/\/digital-strategy.ec.europa.eu\/en\/library\/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence"},{"key":"3435_CR9","doi-asserted-by":"crossref","unstructured":"Ribeiro M T, Singh S, Guestrin C (2016) ``Why should I trust you?\u201d Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"3435_CR10","unstructured":"Kam H T (1995) Random decision forests. In: 3rd International Conference on Document Analysis and Recognition, vol 1416. Montr\u00e9al, Canada, pp 278\u2013282"},{"key":"3435_CR11","doi-asserted-by":"crossref","unstructured":"Haffar R, Domingo-Ferrer J, S\u00e1nchez D (2020) Explaining misclassification and attacks in deep learning via random forests. In: International Conference on Modeling Decisions for Artificial Intelligence-MDAI 2020. Springer, pp 273\u2013285","DOI":"10.1007\/978-3-030-57524-3_23"},{"key":"3435_CR12","unstructured":"McMahan B, Moore E, Ramage D, Hampson S, y Arcas B A (2017) Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics. PMLR, pp 1273\u20131282"},{"key":"3435_CR13","doi-asserted-by":"crossref","unstructured":"Lamport L, Shostak R, Pease M (2019) The Byzantine generals problem. In: Concurrency: the Works of Leslie Lamport, pp 203\u2013226","DOI":"10.1145\/3335772.3335936"},{"key":"3435_CR14","unstructured":"Fang M, Cao X, Jia J, Gong N (2020) Local model poisoning attacks to Byzantine-robust federated learning. In: 29th USENIX Security Symposium (USENIX Security 20), pp 1605\u20131622"},{"key":"3435_CR15","doi-asserted-by":"publisher","first-page":"1452","DOI":"10.1109\/TIFS.2020.3036801","volume":"16","author":"Y Zhong","year":"2020","unstructured":"Zhong Y, Deng W (2020) Towards transferable adversarial attack against deep face recognition. IEEE Trans Inf Forensic Secur 16:1452\u20131466","journal-title":"IEEE Trans Inf Forensic Secur"},{"key":"3435_CR16","doi-asserted-by":"crossref","unstructured":"Taheri R, Javidan R, Shojafar M, Pooranian Z, Miri A, Conti M (2020) On defending against label flipping attacks on malware detection systems. Neural Comput Appl:1\u201320","DOI":"10.1007\/s00521-020-04904-9"},{"key":"3435_CR17","unstructured":"Kairouz P, McMahan H B, Avent B, Bellet A, Bennis M, Bhagoji A N, Bonawitz K, Charles Z, Cormode G, Cummings R (2019) Advances and open problems in federated learning. arXiv:1912.04977"},{"key":"3435_CR18","doi-asserted-by":"crossref","unstructured":"Hitaj B, Ateniese G, Perez-Cruz F (2017) Deep models under the GAN: information leakage from collaborative deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security-CCS\u201917, pp 603\u2013618","DOI":"10.1145\/3133956.3134012"},{"issue":"11","key":"3435_CR19","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1145\/3422622","volume":"63","author":"I Goodfellow","year":"2020","unstructured":"Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63(11):139\u2013144","journal-title":"Commun ACM"},{"key":"3435_CR20","unstructured":"Molnar C (2020) Interpretable machine learning. Lulu.com"},{"key":"3435_CR21","doi-asserted-by":"publisher","first-page":"104041","DOI":"10.1016\/j.compbiomed.2020.104041","volume":"126","author":"PR Magesh","year":"2020","unstructured":"Magesh P R, Myloth R D, Tom R J (2020) An explainable machine learning model for early detection of Parkinson\u2019s disease using LIME on DaTscan imagery. Comput Biol Med 126: 104041","journal-title":"Comput Biol Med"},{"key":"3435_CR22","doi-asserted-by":"crossref","unstructured":"Torcianti A, Matzka S (2021) Explainable artificial intelligence for predictive maintenance applications using a local surrogate model. In: 2021 4th International Conference on Artificial Intelligence for Industries (AI4I). IEEE, pp 86\u201388","DOI":"10.1109\/AI4I51902.2021.00029"},{"key":"3435_CR23","doi-asserted-by":"crossref","unstructured":"Hakkoum H, Idri A, Abnane I (2020) Artificial neural networks interpretation using LIME for breast cancer diagnosis. In: World Conference on Information Systems and Technologies. Springer, pp 15\u201324","DOI":"10.1007\/978-3-030-45697-9_2"},{"key":"3435_CR24","first-page":"1","volume":"11","author":"E Strumbelj","year":"2010","unstructured":"Strumbelj E, Kononenko I (2010) An efficient explanation of individual classifications using game theory. J Mach Learn Res 11:1\u201318","journal-title":"J Mach Learn Res"},{"key":"3435_CR25","doi-asserted-by":"crossref","unstructured":"Turner R (2016) A model explanation system. In: 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, pp 1\u20136","DOI":"10.1109\/MLSP.2016.7738872"},{"issue":"3","key":"3435_CR26","doi-asserted-by":"publisher","first-page":"1350","DOI":"10.1214\/15-AOAS848","volume":"9","author":"B Letham","year":"2015","unstructured":"Letham B, Rudin C, McCormick T H, Madigan D (2015) Interpretable classifiers using rules and Bayesian analysis: building a better stroke prediction model. Ann Appl Stat 9(3):1350\u20131371","journal-title":"Ann Appl Stat"},{"key":"3435_CR27","unstructured":"Singh S, Ribeiro M T, Guestrin C (2016) Programs as black-box explanations. arXiv:1611.07579"},{"key":"3435_CR28","doi-asserted-by":"publisher","first-page":"105532","DOI":"10.1016\/j.knosys.2020.105532","volume":"194","author":"A Blanco-Justicia","year":"2020","unstructured":"Blanco-Justicia A, Domingo-Ferrer J, Mart\u00ednez S, S\u00e1nchez D (2020) Machine learning explainability via microaggregation and shallow decision trees. Knowl-Based Syst 194:105532","journal-title":"Knowl-Based Syst"},{"key":"3435_CR29","unstructured":"Liu Y, Liu Y, Liu Z, Liang Y, Meng C, Zhang J, Zheng Y (2020) Federated forest. IEEE Transactions on Big Data. early access"},{"key":"3435_CR30","volume-title":"Exploratory data analysis, vol 2","author":"JW Tukey","year":"1977","unstructured":"Tukey J W (1977) Exploratory data analysis, vol 2. Reading, MA"},{"key":"3435_CR31","doi-asserted-by":"crossref","unstructured":"Domingo-Ferrer J, Blanco-Justicia A, S\u00e1nchez D, Jebreel N (2020) Co-utile peer-to-peer decentralized computing. In: 20th IEEE\/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID). IEEE, pp 31\u201340","DOI":"10.1109\/CCGrid49817.2020.00-90"},{"key":"3435_CR32","doi-asserted-by":"crossref","unstructured":"Jebreel N, Blanco-Justicia A, S\u00e1nchez D, Domingo-Ferrer J (2020) Efficient detection of Byzantine attacks in federated learning using last layer biases. In: International Conference on Modeling Decisions for Artificial Intelligence-MDAI 2020. Springer, pp 154\u2013165","DOI":"10.1007\/978-3-030-57524-3_13"},{"key":"3435_CR33","doi-asserted-by":"crossref","unstructured":"Cao D, Chang S, Lin Z, Liu G, Sun D (2019) Understanding distributed poisoning attack in federated learning. In: 2019 IEEE 25th International Conference on Parallel and Distributed Systems (ICPADS). IEEE, pp 233\u2013239","DOI":"10.1109\/ICPADS47876.2019.00042"},{"key":"3435_CR34","unstructured":"Blanchard P, El-Mahdi E-M, Guerraoui R, Stainer J (2017) Machine learning with adversaries: Byzantine tolerant gradient descent. In: Advances in Neural Information Processing Systems, pp 119\u2013129"},{"key":"3435_CR35","unstructured":"Yin D, Chen Y, Kannan R, Bartlett P (2018) Byzantine-robust distributed learning: Towards optimal statistical rates. In: International Conference on Machine Learning. PMLR, pp 5650\u20135659"},{"key":"3435_CR36","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.ins.2020.02.037","volume":"522","author":"Y Chen","year":"2020","unstructured":"Chen Y, Luo F, Li T, Xiang T, Liu Z, Li J (2020) A training-integrity privacy-preserving federated learning scheme with trusted execution environment. Inf Sci 522:69\u201379","journal-title":"Inf Sci"},{"issue":"6","key":"3435_CR37","doi-asserted-by":"publisher","first-page":"1279","DOI":"10.1109\/LCOMM.2019.2921755","volume":"24","author":"H Kim","year":"2019","unstructured":"Kim H, Park J, Bennis M, Kim S-L (2019) Blockchained on-device federated learning. IEEE Commun Lett 24(6):1279\u20131283","journal-title":"IEEE Commun Lett"},{"key":"3435_CR38","doi-asserted-by":"publisher","first-page":"10127","DOI":"10.1109\/ACCESS.2018.2890507","volume":"7","author":"K Salah","year":"2019","unstructured":"Salah K, Rehman M H U, Nizamuddin N, Al-Fuqaha A (2019) Blockchain for AI: Review and open research challenges. IEEE Access 7:10127\u201310149","journal-title":"IEEE Access"},{"key":"3435_CR39","doi-asserted-by":"publisher","first-page":"3454","DOI":"10.1109\/TIFS.2020.2988575","volume":"15","author":"K Wei","year":"2020","unstructured":"Wei K, Li J, Ding M, Ma C, Yang HH, Farhad F, Jin S, Quek TQS, Poor V (2020) Federated learning with differential privacy: algorithms and performance analysis. IEEE Trans Inf Forensic Secur 15:3454\u20133469","journal-title":"IEEE Trans Inf Forensic Secur"},{"key":"3435_CR40","doi-asserted-by":"crossref","unstructured":"Bonawitz K, Ivanov V, Kreuter B, Marcedone A, McMahan H B, Patel S, Ramage D, Segal A, Seth K (2017) Practical secure aggregation for privacy-preserving machine learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp 1175\u20131191","DOI":"10.1145\/3133956.3133982"},{"issue":"7","key":"3435_CR41","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1145\/3433638","volume":"64","author":"J Domingo-Ferrer","year":"2021","unstructured":"Domingo-Ferrer J, S\u00e1nchez D, Blanco-Justicia A (2021) The limits of differential privacy (and its misuse in data release and machine learning). Commun ACM 64(7):33\u201335","journal-title":"Commun ACM"},{"key":"3435_CR42","doi-asserted-by":"publisher","first-page":"104468","DOI":"10.1016\/j.engappai.2021.104468","volume":"106","author":"A Blanco-Justicia","year":"2021","unstructured":"Blanco-Justicia A, Domingo-Ferrer J, Mart\u00ednez S, S\u00e1nchez D, Flanagan A, Tan K E (2021) Achieving security and privacy in federated learning systems: survey, research challenges and future directions. Eng Appl Artif Intell 106:104468","journal-title":"Eng Appl Artif Intell"},{"key":"3435_CR43","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: Machine learning in Python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"},{"key":"3435_CR44","doi-asserted-by":"crossref","unstructured":"Reiss A, Stricker D (2012) Introducing a new benchmarked dataset for activity monitoring. In: 16th International Symposium on Wearable Computers. IEEE, pp 108\u2013109","DOI":"10.1109\/ISWC.2012.13"},{"key":"3435_CR45","unstructured":"Gulli A, Pal S (2017) Deep learning with Keras. Packt Publishing Ltd"},{"key":"3435_CR46","unstructured":"Kingma D P, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015. arxiv:1412.6980. Conference Track Proceedings, San Diego"},{"key":"3435_CR47","unstructured":"Lian X, Zhang C, Zhang H, Hsieh C-J, Zhang W, Liu J (2017) Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In: Advances in Neural Information Processing Systems, pp 5330\u20135340"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-022-03435-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-022-03435-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-022-03435-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,3]],"date-time":"2023-01-03T04:32:37Z","timestamp":1672720357000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-022-03435-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,13]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1]]}},"alternative-id":["3435"],"URL":"https:\/\/doi.org\/10.1007\/s10489-022-03435-1","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,13]]},"assertion":[{"value":"23 February 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 April 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}