{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T19:02:19Z","timestamp":1784401339237,"version":"3.55.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T00:00:00Z","timestamp":1613952000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2021,3]]},"abstract":"<jats:p>Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training.<\/jats:p>\n          <jats:p>Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.<\/jats:p>\n          <jats:p>We interpret our experimental findings by comparison with traditional models.<\/jats:p>\n          <jats:p>We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.<\/jats:p>","DOI":"10.1145\/3446776","type":"journal-article","created":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T16:38:41Z","timestamp":1614011921000},"page":"107-115","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2091,"title":["Understanding deep learning (still) requires rethinking generalization"],"prefix":"10.1145","volume":"64","author":[{"given":"Chiyuan","family":"Zhang","sequence":"first","affiliation":[{"name":"Google Brain, Mountain View, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samy","family":"Bengio","sequence":"additional","affiliation":[{"name":"Google Brain, Mountain View, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moritz","family":"Hardt","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Recht","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oriol","family":"Vinyals","sequence":"additional","affiliation":[{"name":"DeepMind, London N1C 4AG, U.K"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,2,22]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"7422","article-title":"Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9 Buc, E. Fox, and R. Garnett, eds. Vol. 32. Curran Associates","volume":"7411","author":"Arora S.","year":"2019","unstructured":"Arora, S., Cohen, N., Hu, W., Luo, Y. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9 Buc, E. Fox, and R. Garnett, eds. Vol. 32. Curran Associates, Inc., 2019, 7411--7422.","journal-title":"Inc."},{"key":"e_1_2_1_2_1","volume-title":"International Conference on Machine Learning. J. Dy and A. Krause, eds.","author":"Arora S.","year":"2018","unstructured":"Arora, S., Ge, R., Neyshabur, B., Zhang, Y. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 254--263."},{"key":"e_1_2_1_3_1","volume-title":"International Conference on Machine Learning. D. Precup and Y.W. Teh, eds.","author":"Arpit D.","year":"2017","unstructured":"Arpit, D., Jastrz\u0119bski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., Lacoste-Julien, S. A closer look at memorization in deep networks. In International Conference on Machine Learning. D. Precup and Y.W. Teh, eds. 2017, 233--242."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/18.661502"},{"key":"e_1_2_1_5_1","first-page":"6249","article-title":"Spectrally-normalized margin bounds for neural networks","volume":"6240","author":"Bartlett P.L.","year":"2017","unstructured":"Bartlett, P.L., Foster, D.J., Telgarsky, M.J. Spectrally-normalized margin bounds for neural networks. Adv. Neural Inform. Process. Syst. 2017, 6240--6249.","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"e_1_2_1_6_1","first-page":"116","article-title":"Reconciling modern machine-learning practice and the classical bias--variance trade-off","volume":"32","author":"Belkin M.","year":"2019","unstructured":"Belkin, M., Hsu, D., Ma, S., Mandal, S. Reconciling modern machine-learning practice and the classical bias--variance trade-off. Proc. Natl. Acad. Sci. 32, 116 (2019), 15849--15854.","journal-title":"Proc. Natl. Acad. Sci."},{"key":"e_1_2_1_7_1","first-page":"2311","article-title":"Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate","volume":"2300","author":"Belkin M.","year":"2018","unstructured":"Belkin, M., Hsu, D.J., Mitra, P. Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate. Adv. Neural Inform. Process. Syst., 2018, 2300--2311.","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1162\/153244302760200704"},{"key":"e_1_2_1_9_1","unstructured":"Choromanska A. Henaff M. Mathieu M. Arous G.B. LeCun Y. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics. G. Lebanon and S.V.N. Vishwanathan eds. 2015 192--204."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35289-8_30"},{"key":"e_1_2_1_11_1","volume-title":"International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds.","author":"Cohen N.","year":"2016","unstructured":"Cohen, N., Shashua, A. Convolutional rectifier networks as generalized tensor decompositions. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 955--963."},{"key":"e_1_2_1_12_1","first-page":"2","article-title":"Approximation by superposition of sigmoidal functions","volume":"4","author":"Cybenko G","year":"1989","unstructured":"Cybenko, G. Approximation by superposition of sigmoidal functions. Math. Control Signal. Syst. 4, 2 (1989), 303--314.","journal-title":"Math. Control Signal. Syst."},{"key":"e_1_2_1_13_1","volume-title":"Advances in Neural Information Processing Systems 24","author":"Delalleau O.","year":"2011","unstructured":"Delalleau, O., Bengio, Y. Shallow vs. deep sum-product networks. In Advances in Neural Information Processing Systems 24. J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, K. Weinberger, eds. Curran Associates, Inc., 2011, 666--674"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence.","author":"Dziugaite G.K.","year":"2017","unstructured":"Dziugaite, G.K., Roy, D.M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence. 2017."},{"key":"e_1_2_1_15_1","volume-title":"Conference on Learning Theory. V. Feldman, A. Rakhlin, and O. Shamir, eds.","author":"Eldan R.","year":"2016","unstructured":"Eldan, R., Shamir, O. The power of depth for feedforward neural networks. In Conference on Learning Theory. V. Feldman, A. Rakhlin, and O. Shamir, eds. 2016, 907--940."},{"key":"e_1_2_1_16_1","volume-title":"Does learning require memorization? a short tale about a long tail. arXiv preprint arXiv:1906.05271","author":"Feldman V.","year":"2019","unstructured":"Feldman V. Does learning require memorization? a short tale about a long tail. arXiv preprint arXiv:1906.05271 (2019)."},{"key":"e_1_2_1_17_1","volume-title":"Conference On Learning Theory. P.R. S\u00e9bastien Bubeck, V. Perchet, eds.","author":"Golowich N.","year":"2018","unstructured":"Golowich, N., Rakhlin, A., Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory. P.R. S\u00e9bastien Bubeck, V. Perchet, eds., 2018, 297--299."},{"key":"e_1_2_1_18_1","volume-title":"International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds.","author":"Hardt M.","year":"2016","unstructured":"Hardt, M., Recht, B., Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 1225--1234."},{"key":"e_1_2_1_19_1","volume-title":"Generalization in deep learning. CoRR, arXiv:1710.05468","author":"Kawaguchi K.","year":"2017","unstructured":"Kawaguchi, K., Kaelbling, L.P., Bengio, Y. Generalization in deep learning. CoRR, arXiv:1710.05468 (2017)."},{"key":"e_1_2_1_20_1","volume-title":"The 22nd International Conference on Artificial Intelligence and Statistics. K. Chaudhuri and M. Sugiyama, eds. arXiv:1711","author":"Liang T.","year":"2017","unstructured":"Liang, T., Poggio, T., Rakhlin, A., Stokes, J. Fisher-rao metric, geometry, and complexity of neural networks. In The 22nd International Conference on Artificial Intelligence and Statistics. K. Chaudhuri and M. Sugiyama, eds. arXiv:1711.01530 (2017), 888--896."},{"key":"e_1_2_1_21_1","volume-title":"International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds.","author":"Lin J.","year":"2016","unstructured":"Lin, J., Camoriano, R., Rosasco, L. Generalization properties and implicit regularization for multiple passes SGM. In International Conference on Machine Learning. M.F. Balcan and K.Q. Weinberger, eds. 2016, 2340--2348."},{"key":"e_1_2_1_22_1","first-page":"863","article-title":"On the computational efficiency of training neural networks. In Advances in Neural Information Processing Systems 27. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, eds. Curran Associates","volume":"855","author":"Livni R.","year":"2014","unstructured":"Livni, R., Shalev-Shwartz, S., Shamir, O. On the computational efficiency of training neural networks. In Advances in Neural Information Processing Systems 27. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, eds. Curran Associates, Inc., 2014, 855--863.","journal-title":"Inc."},{"key":"e_1_2_1_23_1","volume-title":"International Conference on Machine Learning. J. Dy and A. Krause, eds.","author":"Ma S.","year":"2018","unstructured":"Ma, S., Bassily, R., Belkin, M. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning. J. Dy and A. Krause, eds. 2018, 3325--3334."},{"key":"e_1_2_1_24_1","first-page":"14","article-title":"Deep vs. shallow networks: An approximation theory perspective","volume":"6","author":"Mhaskar H.","year":"2016","unstructured":"Mhaskar, H. Poggio, T.A. Deep vs. shallow networks: An approximation theory perspective. Anal. Appl. 6, 14 (2016).","journal-title":"Anal. Appl."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02070821"},{"key":"e_1_2_1_26_1","volume-title":"Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst","author":"Morcos A.","year":"2018","unstructured":"Morcos, A., Raghu, M., Bengio, S. Insights on representational similarity in neural networks with canonical correlation. Adv. Neural Inform. Process. Syst.. 2018, 5727--5736."},{"key":"e_1_2_1_28_1","first-page":"5956","article-title":"Exploring generalization in deep learning","volume":"5947","author":"Neyshabur B.","year":"2017","unstructured":"Neyshabur, B., Bhojanapalli, S., McAllester, D., Srebro, N. Exploring generalization in deep learning. Adv. Neural Inform. Process. Syst., 2017, 5947--5956.","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"e_1_2_1_29_1","volume-title":"International Conference on Learning Representations","author":"Neyshabur B.","year":"2018","unstructured":"Neyshabur, B., Bhojanapalli, S., Srebro, N. A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations, 2018."},{"key":"e_1_2_1_30_1","volume-title":"In search of the real inductive bias: On the role of implicit regularization in deep learning. CoRR, abs\/1412.6614","author":"Neyshabur B.","year":"2014","unstructured":"Neyshabur, B., Tomioka, R., Srebro, N. In search of the real inductive bias: On the role of implicit regularization in deep learning. CoRR, abs\/1412.6614, 2014."},{"key":"e_1_2_1_31_1","volume-title":"Conference on Learning Theory. S.K. Peter Gr\u00fcnwald and E. Hazan, eds.","author":"Neyshabur B.","year":"2015","unstructured":"Neyshabur, B., Tomioka, R., Srebro, N. Norm-based capacity control in neural networks. In Conference on Learning Theory. S.K. Peter Gr\u00fcnwald and E. Hazan, eds. 2015, 1376--1401."},{"key":"e_1_2_1_32_1","first-page":"428","article-title":"General conditions for predictivity in learning theory","volume":"6981","author":"Poggio T.","year":"2004","unstructured":"Poggio, T., Rifkin, R., Mukherjee, S., Niyogi, P. General conditions for predictivity in learning theory. Nature 6981, 428 (2004), 419--422.","journal-title":"Nature"},{"key":"e_1_2_1_33_1","volume-title":"Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811","author":"Recht B.","year":"2019","unstructured":"Recht, B., Roelofs, R., Schmidt, L., Shankar, V. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811 (2019)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-44581-1_27"},{"key":"e_1_2_1_35_1","volume-title":"Minimum norm solutions do not always generalize well for over-parameterized problems. CoRR. arXiv:1811.07055","author":"Shah V.","year":"2018","unstructured":"Shah, V., Kyrillidis, A., Sanghavi, S. Minimum norm solutions do not always generalize well for over-parameterized problems. CoRR. arXiv:1811.07055 (2018)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1953019"},{"key":"e_1_2_1_37_1","volume-title":"International Conference on Learning Representations","author":"Smith S.L.","year":"2018","unstructured":"Smith, S.L., Le, Q.V. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018."},{"key":"e_1_2_1_38_1","first-page":"19","article-title":"The implicit bias of gradient descent on separable data","volume":"70","author":"Soudry D.","year":"2018","unstructured":"Soudry, D., Hoffer, E., Nacson, M.S., Gunasekar, S., Srebro, N. The implicit bias of gradient descent on separable data. J. Mach. Learn. Res. 70, 19 (2018), 1--57.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_39_1","first-page":"15","article-title":"A simple way to prevent neural networks from overfitting","volume":"1","author":"Srivastava N.","year":"2014","unstructured":"Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 1, 15 (2014), 1929--1958.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_40_1","volume-title":"Conference on Learning Theory. V. Feldman, A. Rakhlin, and O. Shamir, eds.","author":"Telgarsky M.","year":"2016","unstructured":"Telgarsky, M. Benefits of depth in neural networks. In Conference on Learning Theory. V. Feldman, A. Rakhlin, and O. Shamir, eds. 2016, 1517--1539."},{"key":"e_1_2_1_41_1","volume-title":"ICLR","author":"Toneva M.","year":"2019","unstructured":"Toneva, M., Sordoni, A., des Combes, R.T., Trischler, A., Bengio, Y., Gordon, G.J. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019."},{"key":"e_1_2_1_42_1","volume-title":"Statistical Learning Theory. Adaptive and Learning Systems for Signal Processing, Communications, and Control","author":"Vapnik V.N.","year":"1998","unstructured":"Vapnik, V.N. Statistical Learning Theory. Adaptive and Learning Systems for Signal Processing, Communications, and Control. Wiley, 1998."},{"key":"e_1_2_1_43_1","first-page":"26","article-title":"On early stopping in gradient descent learning","volume":"2","author":"Yao Y.","year":"2007","unstructured":"Yao, Y., Rosasco, L., Caponnetto, A. On early stopping in gradient descent learning. Const. Approx. 2, 26 (2007), 289--315.","journal-title":"Const. Approx."},{"key":"e_1_2_1_44_1","volume-title":"International Conference on Learning Representations.","author":"Zhang C.","year":"2017","unstructured":"Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations. 2017."},{"key":"e_1_2_1_45_1","first-page":"10801","article-title":"Bias and generalization in deep generative models: An empirical study. In Advances in Neural Information Processing Systems 31. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds. Curran Associates","volume":"10792","author":"Zhao S.","year":"2018","unstructured":"Zhao, S., Ren, H., Yuan, A., Song, J., Goodman, N., Ermon, S. Bias and generalization in deep generative models: An empirical study. In Advances in Neural Information Processing Systems 31. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds. Curran Associates, Inc., 2018, 10792--10801.","journal-title":"Inc."},{"key":"e_1_2_1_46_1","volume-title":"International Conference on Learning Representations","author":"Zhou W.","year":"2019","unstructured":"Zhou, W., Veitch, V., Austern, M., Adams, R.P., Orbanz, P. Non-vacuous generalization bounds at the ImageNet scale: A PAC-Bayesian compression approach. In International Conference on Learning Representations, 2019."}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3446776","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3446776","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:31Z","timestamp":1750193251000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3446776"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,22]]},"references-count":45,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["10.1145\/3446776"],"URL":"https:\/\/doi.org\/10.1145\/3446776","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,22]]},"assertion":[{"value":"2021-02-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}