{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:50:26Z","timestamp":1791323426702,"version":"4.3.1"},"reference-count":53,"publisher":"Institute of Mathematical Statistics","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2026,10,1]]},"abstract":"We provide universality results that quantify how data augmentation affects the variance and limiting distribution of estimates through simple surrogates, and analyze several specific models in detail. The results confirm some observations made in machine learning practice, but also lead to unexpected findings: Data augmentation may increase rather than decrease the uncertainty of estimates, such as the empirical prediction risk. It can act as a regularizer, but fails to do so in certain high-dimensional problems, and it may shift the double-descent peak of an empirical risk. Overall, the analysis shows that several properties attributed to data augmentation are not either true or false, but rather depend on a combination of factors\u2014notably the data distribution, the properties of the estimator, and the interplay of sample size, number of augmentations and dimension. As our main theoretical tool, we develop an adaptation of Lindeberg\u2019s technique for block dependence. The resulting universality regime may be Gaussian or non-Gaussian.<\/jats:p>","DOI":"10.1214\/26-aos2642","type":"journal-article","created":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T12:49:38Z","timestamp":1790945378000},"page":"2325-2351","source":"Crossref","is-referenced-by-count":0,"title":["Gaussian and non-Gaussian universality of data augmentation"],"prefix":"10.1214","volume":"54","author":[{"given":"Kevin Han","family":"Huang","sequence":"first","affiliation":[{"name":"Gatsby Computational Neuroscience Unit, University College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Orbanz","sequence":"additional","affiliation":[{"name":"Gatsby Computational Neuroscience Unit, University College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Morgane","family":"Austern","sequence":"additional","affiliation":[{"name":"Department of Statistics, Harvard University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"11","doi-asserted-by":"publisher","unstructured":"Breiman, L.<\/small> (1996). Bagging predictors. Mach. Learn.<\/i> 24<\/b> 123\u2013140.","DOI":"10.1007\/BF00058655"},{"key":"14","doi-asserted-by":"publisher","unstructured":"Chatterjee, S.<\/small> (2006). A generalization of the Lindeberg principle. Ann. Probab.<\/i> 34<\/b> 2061\u20132076.","DOI":"10.1214\/009117906000000575"},{"key":"42","doi-asserted-by":"publisher","unstructured":"MOSSEL, E.<\/small>, O\u2019DONNELL, R.<\/small> and OLESZKIEWICZ, K.<\/small> (2010). Noise stability of functions with low influences: Invariance and optimality. Ann. of Math<\/i>. (2) 171<\/b> 295\u2013341.","DOI":"10.4007\/annals.2010.171.295"},{"key":"33","doi-asserted-by":"publisher","unstructured":"KORADA, S. B.<\/small> and MONTANARI, A.<\/small> (2011). Applications of the Lindeberg principle in communications and statistical learning. IEEE Trans. Inf. Theory<\/i> 57<\/b> 2440\u20132450.","DOI":"10.1109\/TIT.2011.2112231"},{"key":"27","doi-asserted-by":"publisher","unstructured":"HASTIE, T.<\/small>, MONTANARI, A.<\/small>, ROSSET, S.<\/small> and TIBSHIRANI, R. J.<\/small> (2022). Surprises in high-dimensional ridgeless least squares interpolation. Ann. Statist.<\/i> 50<\/b> 949\u2013986.","DOI":"10.1214\/21-aos2133"},{"key":"38","doi-asserted-by":"publisher","unstructured":"MEI, S.<\/small> and MONTANARI, A.<\/small> (2022). The generalization error of random features regression: Precise asymptotics and the double descent curve. Comm. Pure Appl. Math.<\/i> 75<\/b> 667\u2013766.","DOI":"10.1002\/cpa.22008"},{"key":"7","doi-asserted-by":"publisher","unstructured":"BALLY, V.<\/small> and CARAMELLINO, L.<\/small> (2019). Total variation distance between stochastic polynomials and invariance principles. Ann. Probab.<\/i> 47<\/b> 3762\u20133811.","DOI":"10.1214\/19-aop1346"},{"key":"28","doi-asserted-by":"publisher","unstructured":"HU, H.<\/small> and LU, Y. M.<\/small> (2023). Universality laws for high-dimensional learning with random features. IEEE Trans. Inf. Theory<\/i> 69<\/b> 1932\u20131964.","DOI":"10.1109\/tit.2022.3217698"},{"key":"10","doi-asserted-by":"publisher","unstructured":"Bousquet, O.<\/small> and Elisseeff, A.<\/small> (2002). Stability and generalization. J. Mach. Learn. Res.<\/i> 2<\/b> 499\u2013526.","DOI":"10.1162\/153244302760200704"},{"key":"9","doi-asserted-by":"publisher","unstructured":"Ben Arous, G.<\/small> and Guionnet, A.<\/small> (2008). The spectrum of heavy tailed random matrices. Comm. Math. Phys.<\/i> 278<\/b> 715\u2013751.","DOI":"10.1007\/s00220-007-0389-x"},{"key":"51","doi-asserted-by":"publisher","unstructured":"Tao, T.<\/small> and Vu, V.<\/small> (2011). Random matrices: Universality of local eigenvalue statistics. Acta Math.<\/i> 206<\/b> 127\u2013204.","DOI":"10.1007\/s11511-011-0061-3"},{"key":"32","doi-asserted-by":"publisher","unstructured":"Knowles, A.<\/small> and Yin, J.<\/small> (2017). Anisotropic local laws for random matrices. Probab. Theory Related Fields<\/i> 169<\/b> 257\u2013352.","DOI":"10.1007\/s00440-016-0730-4"},{"key":"45","doi-asserted-by":"publisher","unstructured":"Pillai, N. S.<\/small> and Yin, J.<\/small> (2014). Universality of covariance matrices. Ann. Appl. Probab.<\/i> 24<\/b> 935\u20131001.","DOI":"10.1214\/13-AAP939"},{"key":"37","unstructured":"Mar\u010denko, V. A.<\/small> and Pastur, L. A.<\/small> (1967). Distribution of eigenvalues in certain sets of random matrices. Mat. Sb<\/i>. (N.S<\/i>.) 72<\/b> 507\u2013536."},{"key":"53","doi-asserted-by":"crossref","unstructured":"ZHANG, C.<\/small>, BENGIO, S.<\/small>, HARDT, M.<\/small>, RECHT, B.<\/small> and VINYALS, O.<\/small> (2021). Understanding deep learning (still) requires rethinking generalization. Commun. ACM<\/i> 64<\/b> 107\u2013115.","DOI":"10.1145\/3446776"},{"key":"3","doi-asserted-by":"publisher","unstructured":"AUSTERN, M.<\/small> and ORBANZ, P.<\/small> (2022). Limit theorems for distributions invariant under groups of transformations. Ann. Statist.<\/i> 50<\/b> 1960\u20131991.","DOI":"10.1214\/21-aos2165"},{"key":"29","unstructured":"HUANG, K. H.<\/small>, AUSTERN, M.<\/small> and ORBANZ, P.<\/small> (2024). Gaussian universality for approximately polynomial functions of high-dimensional data. Preprint. Available at arXiv:2403.10711<\/a>."},{"key":"26","doi-asserted-by":"publisher","unstructured":"HAN, Q.<\/small> and SHEN, Y.<\/small> (2023). Universality of regularized regression estimators in high dimensions. Ann. Statist.<\/i> 51<\/b> 1799\u20131823.","DOI":"10.1214\/23-aos2309"},{"key":"41","unstructured":"MONTANARI, A.<\/small> and SAEED, B. N.<\/small> (2022). Universality of empirical risk minimization. In Conference on Learning Theory<\/i> 4310\u20134312. PMLR."},{"key":"49","doi-asserted-by":"crossref","unstructured":"SHORTEN, C.<\/small> and KHOSHGOFTAAR, T. M.<\/small> (2019). A survey on image data augmentation for deep learning. J. Big Data<\/i> 6<\/b> 1\u201348.","DOI":"10.1186\/s40537-019-0197-0"},{"key":"40","unstructured":"MONTANARI, A.<\/small>, RUAN, F.<\/small>, SAEED, B.<\/small> and SOHN, Y.<\/small> (2023). Universality of max-margin classifiers. Preprint. Available at arXiv:2310.00176<\/a>."},{"key":"22","doi-asserted-by":"publisher","unstructured":"DUDEJA, R.<\/small>, LU, Y. M.<\/small> and SEN, S.<\/small> (2023). Universality of approximate message passing with semirandom matrices. Ann. Probab.<\/i> 51<\/b> 1616\u20131683.","DOI":"10.1214\/23-aop1628"},{"key":"8","unstructured":"BAYLE, P.<\/small>, BAYLE, A.<\/small>, JANSON, L.<\/small> and MACKEY, L.<\/small> (2020). Cross-validation confidence intervals for test error. Adv. Neural Inf. Process. Syst.<\/i> 33<\/b> 16339\u201316350."},{"key":"16","doi-asserted-by":"crossref","unstructured":"CHEN, Q.<\/small>, SYRGKANIS, V.<\/small> and AUSTERN, M.<\/small> (2022). Debiased machine learning without sample-splitting for stable estimators. Adv. Neural Inf. Process. Syst.<\/i> 35<\/b> 3096\u20133109.","DOI":"10.52202\/068431-0224"},{"key":"1","unstructured":"ARORA, S.<\/small>, COHEN, N.<\/small>, HU, W.<\/small> and LUO, Y.<\/small> (2019). Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems<\/i> 32<\/b> 7411\u20137422."},{"key":"2","unstructured":"ARORA, S.<\/small>, DU, S. S.<\/small>, HU, W.<\/small>, LI, Z.<\/small>, SALAKHUTDINOV, R. R.<\/small> and WANG, R.<\/small> (2019). On exact computation with an infinitely wide neural net. Adv. Neural Inf. Process. Syst.<\/i> 32<\/b>."},{"key":"4","unstructured":"BA, J.<\/small>, ERDOGDU, M.<\/small>, SUZUKI, T.<\/small>, WU, D.<\/small> and ZHANG, T.<\/small> (2019). Generalization of two-layer neural networks: An asymptotic viewpoint. In International Conference on Learning Representations<\/i>."},{"key":"5","doi-asserted-by":"crossref","unstructured":"BALESTRIERO, R.<\/small>, BOTTOU, L.<\/small> and LECUN, Y.<\/small> (2022). The effects of regularization and data augmentation are class dependent. In Conference on Neural Information Processing Systems<\/i>.","DOI":"10.52202\/068431-2745"},{"key":"6","doi-asserted-by":"crossref","unstructured":"BALESTRIERO, R.<\/small>, MISRA, I.<\/small> and LECUN, Y.<\/small> (2022). A data-augmentation is worth a thousand samples: Exact quantification from analytical augmented sample moments. In Conference on Neural Information Processing Systems<\/i>.","DOI":"10.52202\/068431-1427"},{"key":"12","doi-asserted-by":"publisher","unstructured":"BRYSON, J.<\/small>, VERSHYNIN, R.<\/small> and ZHAO, H.<\/small> (2021). Marchenko-Pastur law with relaxed independence conditions. Random Matrices Theory Appl.<\/i> 10<\/b> Paper No. 2150040, 28 pp.","DOI":"10.1142\/S2010326321500404"},{"key":"13","doi-asserted-by":"crossref","unstructured":"CACOULLOS, T.<\/small> (1982). On upper and lower bounds for the variance of a function of a random variable. Ann. Probab.<\/i> 10<\/b> 799\u2013809.","DOI":"10.1214\/aop\/1176993788"},{"key":"15","unstructured":"CHATTERJI, N. S.<\/small>, LONG, P. M.<\/small> and BARTLETT, P. L.<\/small> (2022). The interplay between implicit bias and benign overfitting in two-layer linear networks. J. Mach. Learn. Res.<\/i> 23<\/b> Paper No. [263], 48 pp."},{"key":"17","unstructured":"CHEN, S.<\/small>, DOBRIBAN, E.<\/small> and LEE, J. H.<\/small> (2020). A group-theoretic framework for data augmentation. J. Mach. Learn. Res.<\/i> 21<\/b> Paper No. 245, 71 pp."},{"key":"18","doi-asserted-by":"publisher","unstructured":"DANDI, Y.<\/small>, STEPHAN, L.<\/small>, KRZAKALA, F.<\/small>, LOUREIRO, B.<\/small> and ZDEBOROV\u00c1, L.<\/small> (2024). Universality laws for Gaussian mixtures in generalized linear models. Advances in Neural Information Processing Systems<\/i> 26<\/b>.","DOI":"10.52202\/075280-2388"},{"key":"19","unstructured":"DAO, T.<\/small>, GU, A.<\/small>, RATNER, A.<\/small>, SMITH, V.<\/small>, DE SA, C.<\/small> and R\u00c9, C.<\/small> (2019). A kernel theory of modern data augmentation. In International Conference on Machine Learning<\/i> 1528\u20131537."},{"key":"20","doi-asserted-by":"publisher","unstructured":"DEYA, A.<\/small> and NOURDIN, I.<\/small> (2014). Invariance principles for homogeneous sums of free random variables. Bernoulli<\/i> 20<\/b> 586\u2013603.","DOI":"10.3150\/12-BEJ498"},{"key":"21","unstructured":"DHIFALLAH, O.<\/small> and LU, Y.<\/small> (2021). On the inherent regularization effects of noise injection during training. In International Conference on Machine Learning<\/i> 2665\u20132675. PMLR."},{"key":"23","unstructured":"FENG, S. Y.<\/small>, GANGAL, V.<\/small>, WEI, J.<\/small>, CHANDAR, S.<\/small>, VOSOUGHI, S.<\/small>, MITAMURA, T.<\/small> and HOVY, E.<\/small> (2021). A survey of data augmentation approaches for NLP. In Findings of Assoc. Comput. Linguist.<\/i> 968\u2013988."},{"key":"24","doi-asserted-by":"publisher","unstructured":"GERACE, F.<\/small>, KRZAKALA, F.<\/small>, LOUREIRO, B.<\/small>, STEPHAN, L.<\/small> and ZDEBOROV\u00c1, L.<\/small> (2024). Gaussian universality of perceptrons with random labels. Phys. Rev. E<\/i> 109<\/b> Paper No. 034305, 18 pp.","DOI":"10.1103\/physreve.109.034305"},{"key":"25","doi-asserted-by":"crossref","unstructured":"GRIGOREVSKI\u012c, N. V.<\/small> and \u0160IGANOV, I. S.<\/small> (1976). Some modifications of the Dudley metric. Zap. Nau\u010dn. Sem. Leningrad. Otdel. Mat. Inst. Steklov<\/i>. (LOMI<\/i>) 61<\/b> 17\u201324.","DOI":"10.1111\/j.1748-5827.1976.tb06547.x"},{"key":"30","doi-asserted-by":"crossref","unstructured":"HUANG, K. H.<\/small>, ORBANZ, P.<\/small> and AUSTERN, M.<\/small> (2026). Supplement to \u201cGaussian and non-Gaussian universality of data augmentation.\u201d https:\/\/doi.org\/10.1214\/26-AOS2642SUPP<\/a>","DOI":"10.1214\/26-AOS2642SUPP"},{"key":"31","doi-asserted-by":"publisher","unstructured":"KIM, B.<\/small> and BARBER, R. F.<\/small> (2023). Black-box tests for algorithmic stability. Inf. Inference<\/i> 12<\/b> Paper No. iaad039, 30 pp.","DOI":"10.1093\/imaiai\/iaad039"},{"key":"34","unstructured":"LEE, J.<\/small>, BAHRI, Y.<\/small>, NOVAK, R.<\/small>, SCHOENHOLZ, S. S.<\/small>, PENNINGTON, J.<\/small> and SOHL-DICKSTEIN, J.<\/small> (2018). Deep neural networks as Gaussian processes. In International Conference on Learning Representations<\/i> (ICLR<\/i>)."},{"key":"35","unstructured":"LUO, Y.<\/small> and BARBER, R. F.<\/small> (2024). Is algorithmic stability testable? A unified framework under computational constraints. Preprint. Available at arXiv:2405.15107<\/a>."},{"key":"36","unstructured":"LYLE, C.<\/small>, VAN DER WILK, M.<\/small>, KWIATKOWSKA, M.<\/small>, GAL, Y.<\/small> and BLOEM-REDDY, B.<\/small> (2019). On the benefits of invariance in neural networks. In Conference on Neural Information Processing Systems<\/i>: Workshop on Machine Learning with Guarantees<\/i>."},{"key":"39","doi-asserted-by":"publisher","unstructured":"MIXON, D. G.<\/small>, PARSHALL, H.<\/small> and PI, J.<\/small> (2022). Neural collapse with unconstrained features. Sampl. Theory Signal Process. Data Anal.<\/i> 20<\/b> Paper No. 11, 13 pp.","DOI":"10.1007\/s43670-022-00027-5"},{"key":"43","unstructured":"NAM, Y.<\/small>, LEE, S. H.<\/small>, DOMIN\u00c9, C. C. J.<\/small>, PARK, Y.<\/small>, LONDON, C.<\/small>, CHOI, W.<\/small>, G\u00d6RING, N. A.<\/small> and LEE, S.<\/small> (2025). Position: Solve layerwise linear models first to understand neural dynamical phenomena (neural collapse, emergence, lazy\/rich regime, and grokking). In International Conference on Machine Learning<\/i>."},{"key":"44","unstructured":"PEREZ, L.<\/small> and WANG, J.<\/small> (2017). The effectiveness of data augmentation in image classification using deep learning. Preprint. Available at arXiv:1712.04621<\/a>."},{"key":"46","doi-asserted-by":"crossref","unstructured":"ROTAR, V. I.<\/small> (1976). Limit theorems for multilinear forms and quasipolynomial functions. Theory Probab. Appl.<\/i> 20<\/b> 512\u2013532.","DOI":"10.1137\/1120058"},{"key":"47","doi-asserted-by":"publisher","unstructured":"ROTAR, V. I.<\/small> (1979). Limit theorems for polylinear forms. J. Multivariate Anal.<\/i> 9<\/b> 511\u2013530.","DOI":"10.1016\/0047-259X(79)90055-1"},{"key":"48","unstructured":"SAXE, A. M.<\/small>, MCCLELLAND, J. L.<\/small> and GANGULI, S.<\/small> (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the International Conference on Learning Representations<\/i> (ICLR<\/i>)."},{"key":"50","unstructured":"SOLOFF, J. A.<\/small>, BARBER, R. F.<\/small> and WILLETT, R.<\/small> (2024). Bagging provides assumption-free stability. J. Mach. Learn. Res.<\/i> 25<\/b> Paper No. [131], 35 pp."},{"key":"52","doi-asserted-by":"crossref","unstructured":"TAQI, A. M.<\/small>, AWAD, A.<\/small>, AL-AZZO, F.<\/small> and MILANOVA, M.<\/small> (2018). The impact of multi-optimizers and data augmentation on TensorFlow convolutional neural network performance. In Proc. of IEEE MIPR<\/i> 140\u2013145.","DOI":"10.1109\/MIPR.2018.00032"}],"container-title":["The Annals of Statistics"],"original-title":[],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:05:36Z","timestamp":1791320736000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-54\/issue-5\/Gaussian-and-non-Gaussian-universality-of-data-augmentation\/10.1214\/26-AOS2642.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,1]]},"references-count":53,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,10,1]]}},"URL":"https:\/\/doi.org\/10.1214\/26-aos2642","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,1]]}}}