{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T22:48:21Z","timestamp":1770677301801,"version":"3.49.0"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T00:00:00Z","timestamp":1676073600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T00:00:00Z","timestamp":1676073600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/501100002790","name":"Canadian Network for Research and Innovation in Machining Technology, Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["RGPIN\/04552-2020"],"award-info":[{"award-number":["RGPIN\/04552-2020"]}],"id":[{"id":"10.13039\/501100002790","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,3]]},"DOI":"10.1007\/s10664-022-10262-y","type":"journal-article","created":{"date-parts":[[2023,2,11]],"date-time":"2023-02-11T11:06:25Z","timestamp":1676113585000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Towards understanding quality challenges of the federated learning for neural networks: a first look from the lens of robustness"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3126-6800","authenticated-orcid":false,"given":"Amin Eslami","family":"Abyane","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Derui","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roberto","family":"Souza","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Ma","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hadi","family":"Hemmati","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,2,11]]},"reference":[{"key":"10262_CR1","unstructured":"Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Man\u00e9 D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Vi\u00e9gas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems, software available from tensorflow.org. https:\/\/www.tensorflow.org\/"},{"key":"10262_CR2","unstructured":"Bagdasaryan E, Veit A, Hua Y, Estrin D, Shmatikov V (2020) How to backdoor federated learning. In: Chiappa S, Calandra R (eds) Proceedings of the 23rd international conference on artificial intelligence and statistics, PMLR, Proceedings of Machine Learning Research, vol 108, pp 2938\u20132948. https:\/\/proceedings.mlr.press\/v108\/bagdasaryan20a.html"},{"key":"10262_CR3","unstructured":"Bhagoji AN, Chakraborty S, Mittal P, Calo SB (2019) Analyzing federated learning through an adversarial lens. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th international conference on machine learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, PMLR, Proceedings of Machine Learning Research, vol 97, pp 634\u2013643. http:\/\/proceedings.mlr.press\/v97\/bhagoji19a.html"},{"key":"10262_CR4","unstructured":"Blanchard P, El Mhamdi EM, Guerraoui R, Stainer J (2017) Machine learning, with adversaries: Byzantine tolerant gradient descent. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30. Curran Associates, Inc.. https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/f4b9ec30ad9f68f89b29639786cb62ef-Paper.pdf"},{"key":"10262_CR5","doi-asserted-by":"publisher","unstructured":"Bonawitz K, Ivanov V, Kreuter B, Marcedone A, McMahan HB, Patel S, Ramage D, Segal A, Seth K (2017) Practical secure aggregation for privacy-preserving machine learning. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, CCS \u201917. https:\/\/doi.org\/10.1145\/3133956.3133982. Association for Computing Machinery, New York, pp 1175\u20131191","DOI":"10.1145\/3133956.3133982"},{"key":"10262_CR6","doi-asserted-by":"publisher","unstructured":"Chai Z, Ali A, Zawad S, Truex S, Anwar A, Baracaldo N, Zhou Y, Ludwig H, Yan F, Cheng Y (2020) Tifl: A tier-based federated learning system. In: Proceedings of the 29th international symposium on high-performance parallel and distributed computing, HPDC \u201920. https:\/\/doi.org\/10.1145\/3369583.3392686. Association for Computing Machinery, New York, pp 125\u2013136","DOI":"10.1145\/3369583.3392686"},{"key":"10262_CR7","unstructured":"Chen X, Liu C, Li B, Lu K, Song D (2017) Targeted backdoor attacks on deep learning systems using data poisoning. arXiv:1712.05526"},{"key":"10262_CR8","unstructured":"Fang M, Cao X, Jia J, Gong NZ (2020) Local model poisoning attacks to byzantine-robust federated learning. In: Capkun S, Roesner F (eds) 29th USENIX Security Symposium, USENIX Security 2020, August 12-14, 2020. https:\/\/www.usenix.org\/conference\/usenixsecurity20\/presentation\/fang. USENIX Association, pp 1605\u20131622"},{"key":"10262_CR9","unstructured":"Fu S, Xie C, Li B, Chen Q (2021) Attack-resistant federated learning with residual-based reweighting. arXiv:1912.11464"},{"key":"10262_CR10","unstructured":"Fung C, Yoon CJM, Beschastnikh I (2020) The limitations of federated learning in Sybil settings. In: Symposium on research in attacks, intrusion, and defenses, RAID"},{"key":"10262_CR11","unstructured":"Goodfellow IJ, Shlens J, Szegedy C (2015) Explaining and harnessing adversarial examples. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 1412.6572"},{"key":"10262_CR12","doi-asserted-by":"publisher","first-page":"47230","DOI":"10.1109\/ACCESS.2019.2909068","volume":"7","author":"T Gu","year":"2019","unstructured":"Gu T, Liu K, Dolan-Gavitt B, Garg S (2019) Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7:47230\u201347244. https:\/\/doi.org\/10.1109\/ACCESS.2019.2909068","journal-title":"IEEE Access"},{"key":"10262_CR13","doi-asserted-by":"publisher","unstructured":"Hu Q, Ma L, Xie X, Yu B, Liu Y, Zhao J (2019) Deepmutation++: A mutation testing framework for deep learning systems. In: 2019 34th IEEE\/ACM international conference on automated software engineering (ASE), pp 1158\u20131161. https:\/\/doi.org\/10.1109\/ASE.2019.00126","DOI":"10.1109\/ASE.2019.00126"},{"key":"10262_CR14","doi-asserted-by":"publisher","unstructured":"Humbatova N, Jahangirova G, Tonella P (2021) Deepcrime: Mutation testing of deep learning systems based on real faults. In: Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2021. https:\/\/doi.org\/10.1145\/3460319.3464825. Association for Computing Machinery, New York, pp 67\u201378","DOI":"10.1145\/3460319.3464825"},{"issue":"1\u20132","key":"10262_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/2200000083","volume":"14","author":"P Kairouz","year":"2021","unstructured":"Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Bhagoji AN, Bonawitz K, Charles Z, Cormode G, Cummings R, D\u2019Oliveira RGL, Eichner H, Rouayheb SE, Evans D, Gardner J, Garrett Z, Gasc\u00f3n A, Ghazi B, Gibbons PB, Gruteser M, Harchaoui Z, He C, He L, Huo Z, Hutchinson B, Hsu J, Jaggi M, Javidi T, Joshi G, Khodak M, Konecn\u00fd J, Korolova A, Koushanfar F, Koyejo S, Lepoint T, Liu Y, Mittal P, Mohri M, Nock R, \u00d6zg\u00fcr A, Pagh R, Qi H, Ramage D, Raskar R, Raykova M, Song D, Song W, Stich SU, Sun Z, Suresh AT, Tram\u00e8r F, Vepakomma P, Wang J, Xiong L, Xu Z, Yang Q, Yu FX, Yu H, Zhao S (2021) Advances and open problems in federated learning. Found Trends\u00ae in Mach Learn 14(1\u20132):1\u2013210. https:\/\/doi.org\/10.1561\/2200000083","journal-title":"Found Trends\u00ae in Mach Learn"},{"key":"10262_CR16","unstructured":"Keras (2021a) keras\/cifar10_cnn.py at master \u22c5 keras-team\/keras \u22c5 github. https:\/\/github.com\/keras-team\/keras\/blob\/master\/examples\/cifar10_cnn.py"},{"key":"10262_CR17","unstructured":"Keras (2021b) Simple mnist convnet. https:\/\/keras.io\/examples\/vision\/mnist_convnet\/"},{"key":"10262_CR18","unstructured":"Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images"},{"key":"10262_CR19","unstructured":"Lai F, Zhu X, Madhyastha HV, Chowdhury M (2021) Oort: Efficient federated learning via guided participant selection. In: Brown AD, Lorch JR (eds) 15th USENIX symposium on operating systems design and implementation, OSDI 2021, July 14-16, 2021, USENIX Association, pp 19\u201335. https:\/\/www.usenix.org\/conference\/osdi21\/presentation\/lai"},{"key":"10262_CR20","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1145\/3335772.3335936","volume-title":"The Byzantine generals problem","author":"L Lamport","year":"2019","unstructured":"Lamport L, Shostak R, Pease M (2019) The Byzantine generals problem. Association for Computing Machinery, New York, pp 203\u2013226. https:\/\/doi.org\/10.1145\/3335772.3335936"},{"key":"10262_CR21","unstructured":"Li T, Hu S, Beirami A, Smith V (2021) Ditto: Fair and robust federated learning through personalization. In: Meila M, Zhang T (eds) Proceedings of the 38th international conference on machine learning, PMLR, Proceedings of Machine Learning Research, vol 139, pp 6357\u20136368. https:\/\/proceedings.mlr.press\/v139\/li21h.html"},{"issue":"3","key":"10262_CR22","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1109\/MSP.2020.2975749","volume":"37","author":"T Li","year":"2020","unstructured":"Li T, Sahu AK, Talwalkar A, Smith V (2020a) Federated learning: Challenges, methods, and future directions. IEEE Sig Process Mag 37 (3):50\u201360. https:\/\/doi.org\/10.1109\/MSP.2020.2975749","journal-title":"IEEE Sig Process Mag"},{"key":"10262_CR23","unstructured":"Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020b) Federated optimization in heterogeneous networks. In: Dhillon IS, Papailiopoulos DS, Sze V (eds) Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. mlsys.org, https:\/\/proceedings.mlsys.org\/book\/316.pdf"},{"key":"10262_CR24","doi-asserted-by":"publisher","unstructured":"Li L, Xu W, Chen T, Giannakis GB, Ling Q (2019) RSA: byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets. In: The 23rd AAAI Conference on Artificial Intelligence, AAAI 2019, The 30st Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. https:\/\/doi.org\/10.1609\/aaai.v33i01.33011544. AAAI Press, pp 1544\u20131551","DOI":"10.1609\/aaai.v33i01.33011544"},{"key":"10262_CR25","doi-asserted-by":"crossref","unstructured":"Lyu L, Yu H, Ma X, Sun L, Zhao J, Yang Q, Yu PS (2020) Privacy and robustness in federated learning: Attacks and defenses. arXiv:2012.06337","DOI":"10.1007\/978-3-030-63076-8_1"},{"key":"10262_CR26","unstructured":"Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A (2018) Towards deep learning models resistant to adversarial attacks. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018. OpenReview.net, Conference track proceedings, https:\/\/openreview.net\/forum?id=rJzIBfZAb"},{"key":"10262_CR27","unstructured":"McMahan B, Moore E, Ramage D, Hampson S, Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: Singh A, Zhu X J (eds) Proceedings of the 20th international conference on artificial intelligence and statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, PMLR, Proceedings of Machine Learning Research, vol 54, pp 1273\u20131282. http:\/\/proceedings.mlr.press\/v54\/mcmahan17a.html"},{"issue":"4","key":"10262_CR28","doi-asserted-by":"publisher","first-page":"869","DOI":"10.1016\/j.nic.2005.09.008","volume":"15","author":"SG Mueller","year":"2005","unstructured":"Mueller SG, Weiner MW, Thal LJ, Petersen RC, Jack C, Jagust W, Trojanowski JQ, Toga AW, Beckett L (2005) The alzheimer\u2019s disease neuroimaging initiative. Neuroimaging Clin N Am 15(4):869","journal-title":"Neuroimaging Clin N Am"},{"key":"10262_CR29","doi-asserted-by":"publisher","unstructured":"Mu\u00f1oz Gonz\u00e1lez L, Biggio B, Demontis A, Paudice A, Wongrassamee V, Lupu EC, Roli F (2017) Towards poisoning of deep learning algorithms with back-gradient optimization. In: Proceedings of the 10th ACM workshop on artificial intelligence and security, AISec \u201917. https:\/\/doi.org\/10.1145\/3128572.3140451. Association for Computing Machinery, New York, pp 27\u201338","DOI":"10.1145\/3128572.3140451"},{"key":"10262_CR30","doi-asserted-by":"publisher","unstructured":"Nishio T, Yonetani R (2019) Client selection for federated learning with heterogeneous resources in mobile edge. In: 2019 IEEE international conference on communications, ICC 2019, May 20-24, 2019. https:\/\/doi.org\/10.1109\/ICC.2019.8761315. IEEE, China, pp 1\u20137","DOI":"10.1109\/ICC.2019.8761315"},{"key":"10262_CR31","unstructured":"Pillutla K, Kakade SM, Harchaoui Z (2019) Robust aggregation for federated learning. arXiv:1912.13445"},{"issue":"3","key":"10262_CR32","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis (IJCV) 115(3):211\u2013252. https:\/\/doi.org\/10.1007\/s11263-015-0816-y","journal-title":"Int J Comput Vis (IJCV)"},{"key":"10262_CR33","doi-asserted-by":"publisher","unstructured":"Shen W, Wan J, Chen Z (2018) Munn: Mutation analysis of neural networks. In: 2018 IEEE international conference on software quality, reliability and security companion (QRS-C), pp 108\u2013115. https:\/\/doi.org\/10.1109\/QRS-C.2018.00032","DOI":"10.1109\/QRS-C.2018.00032"},{"key":"10262_CR34","unstructured":"Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCun Y (eds) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. arXiv:1409.1556"},{"key":"10262_CR35","unstructured":"Sun Z, Kairouz P, Suresh AT, McMahan HB (2019) Can you really backdoor federated learning? arXiv:1911.07963"},{"key":"10262_CR36","unstructured":"Wan CP, Chen Q (2021) Robust federated learning with attack-adaptive aggregation. arXiv:2102.05257"},{"key":"10262_CR37","doi-asserted-by":"publisher","unstructured":"Wang J, Dong G, Sun J, Wang X, Zhang P (2019) Adversarial sample detection for deep neural network through model mutation testing. In: 2019 IEEE\/ACM 41st international conference on software engineering (ICSE), pp 1245\u20131256. https:\/\/doi.org\/10.1109\/ICSE.2019.00126","DOI":"10.1109\/ICSE.2019.00126"},{"key":"10262_CR38","unstructured":"Wang H, Yurochkin M, Sun Y, Papailiopoulos DS, Khazaeni Y (2020) Federated learning with matched averaging. In: 8th international conference on learning representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, https:\/\/openreview.net\/forum?id=BkluqlSFDS"},{"issue":"1","key":"10262_CR39","doi-asserted-by":"publisher","first-page":"100008","DOI":"10.1016\/j.hcc.2021.100008","volume":"1","author":"Q Xia","year":"2021","unstructured":"Xia Q, Ye W, Tao Z, Wu J, Li Q (2021) A survey of federated learning for edge computing: Research problems and solutions. High-Confidence Comput 1(1):100008. https:\/\/doi.org\/10.1016\/j.hcc.2021.100008, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S266729522100009X","journal-title":"High-Confidence Comput"},{"key":"10262_CR40","unstructured":"Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:cs.LG\/1708.07747"},{"key":"10262_CR41","unstructured":"Yin D, Chen Y, Kannan R, Bartlett P (2018) Byzantine-robust distributed learning: Towards optimal statistical rates. In: Dy J, Krause A (eds) Proceedings of the 35th international conference on machine learning, PMLR, Proceedings of Machine Learning Research, vol 80, pp 5650\u20135659. https:\/\/proceedings.mlr.press\/v80\/yin18a.html"},{"key":"10262_CR42","doi-asserted-by":"publisher","unstructured":"Zhao L, Jiang J, Feng B, Wang Q, Shen C, Li Q (2021) Sear: Secure and efficient aggregation for byzantine-robust federated learning. IEEE Trans Dependable Secure Comput :1\u20131. https:\/\/doi.org\/10.1109\/TDSC.2021.3093711","DOI":"10.1109\/TDSC.2021.3093711"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10262-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10262-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10262-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,3]],"date-time":"2023-04-03T07:06:23Z","timestamp":1680505583000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10262-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,11]]},"references-count":42,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["10262"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10262-y","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,11]]},"assertion":[{"value":"31 October 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 February 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interest to declare relevant to this article\u2019s content.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}],"article-number":"44"}}