{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T18:35:05Z","timestamp":1787337305493,"version":"build-2736575974"},"reference-count":54,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"2","funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["787960"],"award-info":[{"award-number":["787960"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM Journal on Mathematics of Data Science"],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Abstract.<\/jats:p>\n                  <jats:p>Adversarial training is a widely used method to improve the robustness of deep neural networks (DNNs) over adversarial perturbations. However, it is empirically observed that adversarial training on overparameterized networks often suffers from robust overfitting: it can achieve almost zero adversarial training error while the robust generalization performance is not promising. In this paper, we provide a theoretical understanding of the question of whether overfitted DNNs in adversarial training can generalize from an approximation viewpoint. Specifically, our main results are threefold: (i) For classification, we prove by construction the existence of infinitely many adversarial training classifiers on overparameterized DNNs that obtain arbitrarily small adversarial training error (overfitting) while achieving good robust generalization error under certain conditions concerning the data quality, data separation, and perturbation level. (ii) Linear overparameterization (meaning that the number of parameters is only slightly larger than the sample size) is enough to ensure such an existence if the target function is smooth enough. (iii) For regression, our results demonstrate that there also exist infinitely many overfitted DNNs with linear overparameterization in adversarial training that can achieve almost optimal rates of convergence for the standard generalization error. Overall, our analysis points out that robust overfitting can be avoided, but the required model capacity will depend on the smoothness of the target function, while a robust generalization gap is inevitable. We hope our analysis will give a better understanding of the mathematical foundations of robustness in DNNs from an approximation view.<\/jats:p>","DOI":"10.1137\/24m1634023","type":"journal-article","created":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T07:56:14Z","timestamp":1776844574000},"page":"225-256","source":"Crossref","is-referenced-by-count":0,"title":["Can Overfitted Deep Neural Networks in Adversarial Training Generalize? An Approximation Viewpoint"],"prefix":"10.1137","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8719-2700","authenticated-orcid":true,"given":"Zhongjie","family":"Shi","sequence":"first","affiliation":[{"name":"Cocorresponding author. School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4133-7921","authenticated-orcid":true,"given":"Fanghui","family":"Liu","sequence":"additional","affiliation":[{"name":"Cocorresponding author. School of Mathematical Sciences Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuan","family":"Cao","sequence":"additional","affiliation":[{"name":"Department of Statistics and Actuarial Science, The University of Hong Kong, Pok Fu Lam Road, Hong Kong."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johan A. K.","family":"Suykens","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, ESAT-STADIUS, KU Leuven, Kasteelpark Arenberg 10, B-3001 Leuven, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2026,4,22]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1137\/21M1467134"},{"key":"ref2","doi-asserted-by":"crossref","first-page":"30063","DOI":"10.1073\/pnas.1907378117","volume":"117","author":"Bartlett P. L.","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref3","doi-asserted-by":"crossref","first-page":"15849","DOI":"10.1073\/pnas.1903070116","volume":"116","author":"Belkin M.","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref4","unstructured":"A. N. Bhagoji, D. Cullina, and P. Mittal, Lower bounds on adversarial robustness from optimal transport, in Proceedings of the 33rd International Conference on Neural Information Processing Systems, Adv. Neural Inform. Process. Syst. 32, Curran Associates, 2019, 674, https:\/\/dl.acm.org\/doi\/10.5555\/3454287.3454961."},{"key":"ref5","doi-asserted-by":"crossref","unstructured":"Y. Cao, Z. Chen, M. Belkin, and Q. Gu, Benign overfitting in two-layer convolutional neural networks, in Proceedings of the 36th International Conference on Neural Information Processing Systems, Adv. Neural Inform. Process. Syst. 35, Curran Associates, 2022, pp. 25237\u201325250, https:\/\/dl.acm.org\/doi\/10.5555\/3600270.3602100.","DOI":"10.52202\/068431-1830"},{"key":"ref6","unstructured":"Y. Cao, Q. Gu, and M. Belkin, Risk bounds for over-parameterized maximum margin classification on sub-Gaussian mixtures, in Proceedings of the 35th International Conference on Neural Information Processing Systems, Adv. Neural Inform. Process. Syst. 34, Curran Associates, 2021, pp. 8407\u20138418, https:\/\/dl.acm.org\/doi\/10.5555\/3540261.3540904."},{"key":"ref7","doi-asserted-by":"crossref","unstructured":"N. Carlini and D. Wagner, Adversarial examples are not easily detected: Bypassing ten detection methods, in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 2017, pp. 3\u201314, https:\/\/dl.acm.org\/doi\/10.1145\/3128572.3140444.","DOI":"10.1145\/3128572.3140444"},{"key":"ref8","unstructured":"J. Chen, Y. Cao, and Q. Gu, Benign Overfitting in Adversarially Robust Linear Classification, preprint, arXiv:2112.15250, 2021."},{"key":"ref9","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1090\/S0025-5718-1994-1240656-2","volume":"63","author":"Chui C. K.","year":"1994","journal-title":"Math. Comp."},{"key":"ref10","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1109\/TNNLS.2020.3027613","volume":"33","author":"Chui C. K.","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref11","series-title":"Cambridge Monogr. Appl. Comput. Math. 24","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511618796","volume-title":"Learning Theory: An Approximation Theory Viewpoint","author":"Cucker F.","year":"2007"},{"key":"ref12","unstructured":"C. Dan, Y. Wei, and P. Ravikumar, Sharp statistical guaratees for adversarially robust gaussian classification, in Proceedings of the 37th\u00a0International Conference on Machine Learning, Online, PMLR 119, 2020, pp. 2345\u20132355, https:\/\/proceedings.mlr.press\/v119\/dan20b.html."},{"key":"ref13","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1137\/21M1426286","volume":"4","author":"Delgosha P.","year":"2022","journal-title":"SIAM J. Math. Data Sci."},{"key":"ref14","unstructured":"G. W. Ding, K. Y. C. Lui, X. Jin, L. Wang, and R. Huang, On the sensitivity of adversarial robustness to input data distributions, in Proceedings of the 7th International Conference on Learning Representations (ICLR 2019)., Vol. 2, Curran Associates,\u00a02019, pp. 1056\u20131074, https:\/\/openreview.net\/pdf?id=S1xNEhR9KX."},{"key":"ref15","unstructured":"C. Dong, L. Liu, and J. Shang, Data Quality Matters for Adversarial Training: An Empirical Study, preprint, arXiv:2102.07437, 2021."},{"key":"ref16","doi-asserted-by":"crossref","unstructured":"C. Dong, L. Liu, and J. Shang, Label noise in adversarial training: A novel perspective to study robust overfitting, in Proceedings of the 36th Conference on Neural Information Processing System (NeurIPS 2022), Adv. Neural Inform. Process. Syst. 35, Curran Associates, 2022, pp. 17556\u201317567, https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/6fe6a2ba2594521d15af3b1f2162d79c-Paper-Conference.pdf.","DOI":"10.52202\/068431-1276"},{"key":"ref17","unstructured":"Y. Dong, K. Xu, X. Yang, T. Pang, Z. Deng, H. Su, and J. Zhu, Exploring memorization in adversarial training, in Proceedings of the 10th International Conference on Learning Representations, Curran Associates, 2022, https:\/\/openreview.net\/pdf\/c33e24a96109ee75779a6d6cbb2bb2f7df3ddadd.pdf."},{"key":"ref18","series-title":"Adapt. Comput. Mach. Learn.","volume-title":"Deep Learning","author":"Goodfellow I.","year":"2016"},{"key":"ref19","unstructured":"I. J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples, in Proceedings of the 3rd International Conference on Learning Representations, 2015, https:\/\/arxiv.org\/abs\/1412.6572."},{"key":"ref20","unstructured":"S. Gu and L. Rigazio, Towards Deep Neural Network Architectures Robust to Adversarial Examples, preprint, arXiv:1412.5068, 2014."},{"key":"ref21","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1142\/S0219530519410021","volume":"18","author":"G\u00fchring I.","year":"2020","journal-title":"Anal. Appl."},{"key":"ref22","unstructured":"C. Guo, M. Rana, M. Cisse, and L. van der Maaten, Countering Adversarial Images Using Input Transformations, preprint, arXiv:1711.00117, 2017."},{"key":"ref23","series-title":"Springer Ser. Statist. 1","doi-asserted-by":"crossref","DOI":"10.1007\/b97848","volume-title":"A Distribution-Free Theory of Nonparametric Regression","author":"Gy\u00f6rfi L.","year":"2002"},{"key":"ref24","doi-asserted-by":"crossref","first-page":"1853","DOI":"10.1109\/TPAMI.2020.3032422","volume":"44","author":"Han Z.","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref25","unstructured":"A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, Adversarial examples are not bugs, they are features, in Proceedings of the 33rd International Conference on Neural Information Processing Systems, Curran Associates, 2019, pp. 125\u2013136, https:\/\/dl.acm.org\/doi\/10.5555\/3454287.3454299."},{"key":"ref26","unstructured":"J. Khim and P.L. Loh, Adversarial Risk Bounds via Function Transformation, preprint, arXiv:1810.09519, 2018."},{"key":"ref27","doi-asserted-by":"crossref","unstructured":"B. Li, J. Jin, H. Zhong, J. Hopcroft, and L. Wang, Why robust generalization in deep learning is difficult: Perspective of expressive power, in Proceedings of the 36th International Conference on Neural Information Processing Systems, Adv. Neural Inform. Process. Syst. 35, Curran Associates, 2022, pp. 4370\u20134384, https:\/\/dl.acm.org\/doi\/10.5555\/3600270.3600586.","DOI":"10.52202\/068431-0316"},{"key":"ref28","unstructured":"B. Li and Y. Li, Why Clean Generalization and Robust Overfitting Both Happen in Adversarial Training, preprint, arXiv:2306.01271, 2023."},{"key":"ref29","unstructured":"S.B. Lin, Y. Wang, and D.X. Zhou, Generalization Performance of Empirical Risk Minimization on Over-Parameterized Deep ReLU Nets, preprint, arXiv:2111.14039, 2021."},{"key":"ref30","unstructured":"H. Liu, M. Chen, S. Er, W. Liao, T. Zhang, and T. Zhao, Benefits of overparameterized convolutional residual networks: Function approximation under smoothness constraint, in Proceedings of the 39 th International Conference on Machine Learning, Baltimore, Maryland, USA, PMLR 162, I 2022, pp. 13669\u201313703, https:\/\/proceedings.mlr.press\/v162\/liu22c\/liu22c.pdf."},{"key":"ref31","unstructured":"A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, Towards deep learning models resistant to adversarial attacks, in Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 2018, https:\/\/openreview.net\/pdf?id=rJzIBfZAb."},{"key":"ref32","doi-asserted-by":"crossref","unstructured":"J. X. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi, TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, preprint, arXiv:2005.05909, 2020.","DOI":"10.18653\/v1\/2020.emnlp-demos.16"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1137\/22M1478835"},{"key":"ref34","unstructured":"A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang, Adversarial Training Can Hurt Generalization, preprint, arXiv:1906.06032, 2019."},{"key":"ref35","unstructured":"S.A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. A. Mann, Data augmentation can improve robustness, in Proceedings of the 35th International Conference on Neural Information Processing Systems,\u00a0Adv. Neural Inform. Process. Syst. 34, Curran Associates, 2021, pp. 29935\u201329948, https:\/\/dl.acm.org\/doi\/10.5555\/3540261.3542552."},{"key":"ref36","unstructured":"L. Rice, E. Wong, and Z. Kolter, Overfitting in adversarially robust deep learning, in Proceedings of the 37th International Conference on Machine Learning, PMLR, 2020, pp. 8093\u20138104, https:\/\/dl.acm.org\/doi\/abs\/10.5555\/3524938.3525687."},{"key":"ref37","unstructured":"L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry, Adversarially robust generalization requires more data, in Proceedings of the 32nd International Conference on Neural Information Processing Systems, Adv. Neural Inform. Process. Syst. 31, Curran Associates,\u00a02018, pp. 5019\u20135031, https:\/\/dl.acm.org\/doi\/10.5555\/3327345.3327409."},{"key":"ref38","first-page":"1875","volume":"48","author":"Schmidt-Hieber J.","year":"2020","journal-title":"Ann. Statist."},{"key":"ref39","unstructured":"A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein, Are adversarial examples inevitable? in Proceedings of the 7th\u00a0International Conference on Learning Representations, 2019, https:\/\/openreview.net\/forum?id=r1lWUoA9FQ."},{"key":"ref40","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1016\/j.acha.2012.05.001","volume":"34","author":"Shi L.","year":"2013","journal-title":"Appl. Comput. Harmon. Anal."},{"key":"ref41","doi-asserted-by":"crossref","unstructured":"D. Su, H. Zhang, H. Chen, J. Yi, P.Y. Chen, and Y. Gao, Is robustness the cost of accuracy?\u2014A comprehensive study on the robustness of 18 deep image classification models, in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 631\u2013648, https:\/\/openaccess.thecvf.com\/content_ECCV_2018\/papers\/Dong_Su_Is_Robustness_the_ECCV_2018_paper.pdf.","DOI":"10.1007\/978-3-030-01258-8_39"},{"key":"ref42","unstructured":"C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, Intriguing Properties of Neural Networks, preprint, arXiv:1312.6199, 2013."},{"key":"ref43","first-page":"1","volume":"24","author":"Tsigler A.","year":"2023","journal-title":"J. Mach. Learn. Res."},{"key":"ref44","unstructured":"D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, Robustness may be at odds with accuracy, in International Conference on Learning Representations, 2019, https:\/\/openreview.net\/forum?id=SyxAb30cY7*."},{"key":"ref45","series-title":"Cambridge Monogr. Appl. Comput. Math. 17","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511617539","volume-title":"Scattered Data Approximation","author":"Wendland H.","year":"2004"},{"key":"ref46","unstructured":"J. Xiao, Y. Fan, R. Sun, and Z.Q. Luo, Adversarial Rademacher Complexity of Deep Neural Networks, preprint, arXiv:2211.14966, 2022."},{"key":"ref47","unstructured":"Y.Y. Yang, C. Rashtchian, H. Zhang, R. R. Salakhutdinov, and K. Chaudhuri, A closer look at accuracy vs. robustness, in Proceedings of the 34th International Conference on Neural Information Processing Systems, Adv. Neural Inform. Process. Syst. 33, Curran Associates,\u00a02020, pp. 8588\u20138601, https:\/\/dl.acm.org\/doi\/10.5555\/3495724.3496444."},{"key":"ref48","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.neunet.2017.07.002","volume":"94","author":"Yarotsky D.","year":"2017","journal-title":"Neural Networks"},{"key":"ref49","unstructured":"D. Yin, K. Ramchandran, and P. Bartlett, Rademacher complexity for adversarially robust generalization, in International Conference on Machine Learning, PMLR, 2019, pp. 7085\u20137094, https:\/\/proceedings.mlr.press\/v97\/yin19b\/yin19b-supp.pdf."},{"key":"ref50","doi-asserted-by":"crossref","unstructured":"S. Zagoruyko and N. Komodakis, Wide Residual Networks, preprint, arXiv:1605.07146, 2016.","DOI":"10.5244\/C.30.87"},{"key":"ref51","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/3446776","volume":"64","author":"Zhang C.","year":"2021","journal-title":"Commun. ACM"},{"key":"ref52","unstructured":"H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, Theoretically principled trade-off between robustness and accuracy, in Proceedings of the 36th\u00a0International Conference on Machine Learning, PMLR 97, 2019, pp. 7472\u20137482, https:\/\/proceedings.mlr.press\/v97\/zhang19p\/zhang19p.pdf."},{"key":"ref53","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1214\/aos\/1079120130","volume":"32","author":"Zhang T.","year":"2004","journal-title":"Ann. Statist."},{"key":"ref54","unstructured":"Z. Zhu, F. Liu, G. Chrysos, F. Locatello, and V. Cevher, Benign overfitting in deep neural networks under lazy training, in Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 2023, pp. 43105\u201343128, https:\/\/proceedings.mlr.press\/v202\/zhu23h\/zhu23h.pdf."}],"container-title":["SIAM Journal on Mathematics of Data Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/24M1634023","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T17:56:05Z","timestamp":1787334965000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/24M1634023"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,22]]},"references-count":54,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1137\/24M1634023"],"URL":"https:\/\/doi.org\/10.1137\/24m1634023","relation":{},"ISSN":["2577-0187"],"issn-type":[{"value":"2577-0187","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,22]]}}}