{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T07:55:56Z","timestamp":1793346956138,"version":"4.5.0"},"reference-count":67,"publisher":"National Academy of Sciences","issue":"43","license":[{"start":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T00:00:00Z","timestamp":1634774400000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2021,10,26]]},"abstract":"<jats:title>Significance<\/jats:title>\n                  <jats:p>The remarkable development of deep learning over the past decade relies heavily on sophisticated heuristics and tricks. To better exploit its potential in the coming decade, perhaps a rigorous framework for reasoning about deep learning is needed, which, however, is not easy to build due to the intricate details of neural networks. For near-term purposes, a practical alternative is to develop a mathematically tractable surrogate model, yet maintaining many characteristics of neural networks. This paper proposes a model of this kind that we term the Layer-Peeled Model. The effectiveness of this model is evidenced by, among others, its ability to reproduce a known empirical pattern and to predict a hitherto-unknown phenomenon when training deep-learning models on imbalanced datasets.<\/jats:p>","DOI":"10.1073\/pnas.2103091118","type":"journal-article","created":{"date-parts":[[2021,10,21]],"date-time":"2021-10-21T16:34:09Z","timestamp":1634834049000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":83,"title":["Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training"],"prefix":"10.1073","volume":"118","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5076-7897","authenticated-orcid":false,"given":"Cong","family":"Fang","sequence":"first","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hangfeng","family":"He","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0660-5230","authenticated-orcid":false,"given":"Qi","family":"Long","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, PA 19104;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1787-1219","authenticated-orcid":false,"given":"Weijie J.","family":"Su","sequence":"additional","affiliation":[{"name":"Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA 19104"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2021,10,20]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2015509117"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(90)90019-H"},{"key":"e_1_3_4_7_2","first-page":"2822","article-title":"The implicit bias of gradient descent on separable data","volume":"19","author":"Soudry D.","year":"2018","unstructured":"D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, N. Srebro, The implicit bias of gradient descent on separable data. J. Mach. Learn. Res. 19, 2822\u20132878 (2018).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSAIT.2020.2991332"},{"key":"e_1_3_4_9_2","first-page":"9422","article-title":"Learning diverse and discriminative representations via the principle of maximal coding rate reduction","volume":"33","author":"Yu Y.","year":"2020","unstructured":"Y. Yu, K. H. R. Chan, C. You, C. Song, Y. Ma, Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Adv. Neural Inf. Process. Syst.33, 9422\u20139434 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_10_2","unstructured":"O. Shamir Gradient methods never overfit on separable data. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2007.00028 (Accessed 20 December 2020)."},{"key":"e_1_3_4_11_2","first-page":"1597","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Chen T.","year":"2020","unstructured":"T. Chen, S. Kornblith, M. Norouzi, G. Hinton, \u201cA simple framework for contrastive learning of visual representations\u201d in Proceedings of the 37th International Conference on Machine Learning, H. Daum\u00e9 III, A. Singh, Eds. (Proceedings of Machine Learning Research, Vienna, Austria, 2020), vol. 119, pp. 1597\u20131607."},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0192-5"},{"key":"e_1_3_4_13_2","unstructured":"A. Krizhevsky \u201cLearning multiple layers of features from tiny images \u201d Master\u2019s thesis University of Toronto Toronto Canada (2009)."},{"key":"e_1_3_4_14_2","volume-title":"ICLR 2015: 3rd International Conference on Learning Representations","author":"Simonyan K.","year":"2015","unstructured":"K. Simonyan, A. Zisserman, \u201cVery deep convolutional networks for large-scale image recognition\u201d in ICLR 2015: 3rd International Conference on Learning Representations, Y. Bengio, Y. LeCun, Eds. (ICLR, San Diego, CA, 2015)."},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2017.07.002"},{"key":"e_1_3_4_16_2","first-page":"8580","volume-title":"NeurIPS 2018: 32nd Conference on Neural Information Processing Systems","author":"Jacot A.","year":"2018","unstructured":"A. Jacot, F. Gabriel, C. Hongler, \u201cNeural tangent kernel: Convergence and generalization in neural networks\u201d in NeurIPS 2018: 32nd Conference on Neural Information Processing Systems, S. Bengio ., Eds. (Curran Associates, Inc., Montreal, Canada, 2018), vol. 31, pp. 8580\u20138589."},{"key":"e_1_3_4_17_2","first-page":"1675","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Du S. S.","year":"2019","unstructured":"S. S. Du, J. D. Lee, H. Li, L. Wang, X. Zhai, \u201cGradient descent finds global minima of deep neural networks\u201d in Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri, R. Salakhutdinov, Eds. (Proceedings of Machine Learning Research, Long Beach, CA, 2019), pp. 1675\u20131685."},{"key":"e_1_3_4_18_2","first-page":"2388","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Allen-Zhu Z.","year":"2019","unstructured":"Z. Allen-Zhu, Y. Li, Z. Song, \u201cA convergence theory for deep learning via over-parameterization\u201d in Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri, R. Salakhutdinov, Eds. (Proceedings of Machine Learning Research, Long Beach, CA, 2019), pp. 2388\u20132464."},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05839-6"},{"key":"e_1_3_4_20_2","first-page":"2937","volume-title":"Advances in Neural Information Processing Systems","author":"Chizat L.","year":"2019","unstructured":"L. Chizat, E. Oyallon, F. Bach, \u201cOn lazy training in differentiable programming\u201d in Advances in Neural Information Processing Systems, H. Wallach ., Eds. (Curran Associates, Inc., Vancouver, Canada, 2019), pp. 2937\u20132947."},{"key":"e_1_3_4_21_2","unstructured":"E. Weinan C. Ma L. Wu A comparative analysis of the optimization and generalization property of two-layer neural network and random feature models under gradient descent dynamics. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/1904.04326 (Accessed 21 December 2020)."},{"key":"e_1_3_4_22_2","first-page":"6241","article-title":"Spectrally-normalized margin bounds for neural networks","volume":"30","author":"Bartlett P.","year":"2017","unstructured":"P. Bartlett, D. Foster, M. Telgarsky, Spectrally-normalized margin bounds for neural networks. Adv. Neural Inf. Process. Syst. 30, 6241\u20136250 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_23_2","volume-title":"International Conference on Learning Representations","author":"He H.","year":"2020","unstructured":"H. He, W. J. Su, \u201cThe local elasticity of neural networks\u201d in International Conference on Learning Representations (OpenReview.net, Addis Ababa, Ethiopia, 2020)."},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1907369117"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1806579115"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.spa.2019.06.003"},{"key":"e_1_3_4_27_2","first-page":"7146","volume-title":"Advances in Neural Information Processing Systems","author":"Rotskoff G. M.","year":"2018","unstructured":"G. M. Rotskoff, E. Vanden-Eijnden, \u201cParameters as interacting particles: long time convergence and asymptotic error scaling of neural networks,\u201d in Advances in Neural Information Processing Systems, S. Bengio ., Eds. (Curran Associates, Inc., Montreal, Canada, 2018), pp. 7146\u20137155."},{"key":"e_1_3_4_28_2","unstructured":"C. Fang J. D. Lee P. Yang T. Zhang Modeling from features: A mean-field framework for over-parameterized deep neural networks. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2007.01452 (Accessed 23 December 2020)."},{"key":"e_1_3_4_29_2","first-page":"14601","volume-title":"Advances in Neural Information Processing Systems","author":"Kuditipudi R.","year":"2019","unstructured":"R. Kuditipudi ., \u201cExplaining landscape connectivity of low-cost solutions for multilayer nets\u201d in Advances in Neural Information Processing Systems H. Wallach ., Eds. (Curran Associates, Inc., Vancouver, Canada, 2019), pp. 14601\u201314610."},{"key":"e_1_3_4_30_2","unstructured":"B. Shi W. J. Su M. I. Jordan On learning rates and Schr\u00f6dinger operators. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2004.06977 (Accessed 26 December 2020)."},{"key":"e_1_3_4_31_2","doi-asserted-by":"crossref","unstructured":"C. Fang H. Dong T. Zhang Mathematical models of overparameterized neural networks. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2012.13982 (Accessed 5 January 2021).","DOI":"10.1109\/JPROC.2020.3048020"},{"key":"e_1_3_4_32_2","unstructured":"F. He D. Tao Recent advances in deep learning theory. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2012.10931 (Accessed 5 January 2021)."},{"key":"e_1_3_4_33_2","unstructured":"J. Fan C. Ma Y. Zhong A selective overview of deep learning. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1904.05526 (Accessed 26 December 2020)."},{"key":"e_1_3_4_34_2","unstructured":"R. Sun Optimization for deep learning: Theory and algorithms. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1912.08957 (Accessed 26 December 2020)."},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1063-5203(03)00023-X"},{"key":"e_1_3_4_36_2","first-page":"3325","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Ma S.","year":"2018","unstructured":"S. Ma, R. Bassily, M. Belkin, \u201cThe power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning\u201d in Proceedings of the 35th International Conference on Machine Learning J. Dy, A. Krause, Eds. (Proceedings of Machine Learning Research, Stockholm, Sweden, 2018), vol. 80, pp. 3325\u20133334."},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1903070116"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1214\/19-AOS1849"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1907378117"},{"key":"e_1_3_4_40_2","unstructured":"Z. Li W. Su D. Sejdinovic Benign overfitting and noisy features. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2008.02901 (Accessed 27 December 2020)."},{"key":"e_1_3_4_41_2","unstructured":"D. G. Mixon H. Parshall J. Pi Neural collapse with unconstrained features. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2011.11619 (Accessed 27 December 2020)."},{"key":"e_1_3_4_42_2","unstructured":"E. Weinan S. Wojtowytsch On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2012.05420 (Accessed 6 January 2021)."},{"key":"e_1_3_4_43_2","unstructured":"J. Lu S. Steinerberger Neural collapse with cross-entropy loss. arXiv [Preprint] (2021). https:\/\/arxiv.org\/abs\/2012.08465 (Accessed 6 January 2021)."},{"key":"e_1_3_4_44_2","unstructured":"T. Ergen M. Pilanci Convex duality of deep neural networks. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2002.09773 (Accessed 27 December 2020)."},{"key":"e_1_3_4_45_2","unstructured":"T. Poggio Q. Liao Explicit regularization and implicit bias in deep network classifiers trained with the square loss. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2101.00072 (Accessed 10 January 2021)."},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_4_47_2","first-page":"5628","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Saunshi N.","year":"2019","unstructured":"N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, H. Khandeparkar, \u201cA theoretical analysis of contrastive unsupervised representation learning\u201d in Proceedings of the 36th International Conference on Machine Learning, K. Chaudhuri, R. Salakhutdinov, Eds. (Proceedings of Machine Learning Research, Long Beach, CA, 2019), vol. 97, pp. 5628\u20135637."},{"key":"e_1_3_4_48_2","unstructured":"A. Baevski H. Zhou A. Mohamed M. Auli wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2006.11477 (Accessed 29 December 2020)."},{"key":"e_1_3_4_49_2","unstructured":"P. Khosla . Supervised contrastive learning. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2004.11362 (Accessed 29 December 2020)."},{"key":"e_1_3_4_50_2","first-page":"8026","volume-title":"Advances in Neural Information Processing Systems","author":"Paszke A.","year":"2019","unstructured":"A. Paszke ., \u201cPytorch: An imperative style, high-performance deep learning library\u201d in Advances in Neural Information Processing Systems H. Wallach ., Eds. (Curran Associates, Inc., Vancouver, Canada, 2019), pp. 8026\u20138037."},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2016.7727770"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.580"},{"key":"e_1_3_4_53_2","first-page":"2267","article-title":"Data imbalance and classifiers: Impact and solutions from a big data perspective","volume":"13","author":"Madasamy K.","year":"2017","unstructured":"K. Madasamy, M. Ramaswami, Data imbalance and classifiers: Impact and solutions from a big data perspective. Int. J. Comput. Intell. Res 13, 2267\u20132281 (2017).","journal-title":"Int. J. Comput. Intell. Res"},{"key":"e_1_3_4_54_2","unstructured":"B. Zhou A. Khosla A. Lapedriza A. Torralba A. Oliva Places: An image database for deep scene understanding. arXiv [Preprint] (2016). https:\/\/arxiv.org\/abs\/1610.02055 (Accessed 2 January 2021)."},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.3115\/1614049.1614064"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1287\/moor.28.2.246.14485"},{"key":"e_1_3_4_57_2","unstructured":"F. Bach J. Mairal J. Ponce Convex sparse matrix factorizations. arXiv [Preprint] (2008). https:\/\/arxiv.org\/abs\/0812.1869 (Accessed 3 January 2021)."},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2900306"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_4_60_2","unstructured":"H. Xiao K. Rasul R. Vollgraf Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv [Preprint] (2017). https:\/\/arxiv.org\/abs\/1708.07747 (Accessed 3 January 2021)."},{"key":"e_1_3_4_61_2","first-page":"448","volume-title":"International Conference on Machine Learning","author":"Ioffe S.","year":"2015","unstructured":"S. Ioffe, C. Szegedy, \u201cBatch normalization: Accelerating deep network training by reducing internal covariate shift\u201d in International Conference on Machine Learning, F. Bach, D. Blei, Eds. (PMLR, Lille, France, 2015), pp. 448\u2013456."},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2018.07.011"},{"key":"e_1_3_4_63_2","first-page":"1919","volume-title":"NeurIPS 2019: 33rd International Conference on Neural Information Processing Systems","author":"Shu J.","year":"2019","unstructured":"J. Shu ., \u201cMeta-weight-net: Learning an explicit mapping for sample weighting\u201d in NeurIPS 2019: 33rd International Conference on Neural Information Processing Systems, H. Wallach ., Eds. (Advances in Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, 2019), vol. 32, pp. 1919\u20131930."},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00949"},{"key":"e_1_3_4_65_2","first-page":"1567","volume-title":"NeurIPS 2019: 33rd International Conference on Neural Information Processing Systems","author":"Cao K.","unstructured":"K. Cao, C. Wei, A. Gaidon, N. Arechiga, T. Ma, \u201cLearning imbalanced datasets with label-distribution-aware margin loss\u201d in NeurIPS 2019: 33rd International Conference on Neural Information Processing Systems, H. Wallach ., Eds. (Advances in Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, pp. 1567\u20131578."},{"key":"e_1_3_4_66_2","first-page":"77","volume-title":"Proceedings of the First Conference on Fairness, Accountability and Transparency","author":"Buolamwini J.","year":"2018","unstructured":"J. Buolamwini, T. Gebru, \u201cGender shades: Intersectional accuracy disparities in commercial gender classification\u201d in Proceedings of the First Conference on Fairness, Accountability and Transparency S. A. Friedler, C. Wilson, Eds. (Proceedings of Machine Learning Research, New York, NY, 2018), vol. 81, pp. 77\u201391."},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-018-05707-8"},{"key":"e_1_3_4_68_2","unstructured":"N. Mehrabi F. Morstatter N. Saxena K. Lerman A. Galstyan A survey on bias and fairness in machine learning. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1908.09635 (Accessed 4 January 2021)."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2103091118","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T08:07:06Z","timestamp":1649837226000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/full\/10.1073\/pnas.2103091118"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,20]]},"references-count":67,"journal-issue":{"issue":"43","published-print":{"date-parts":[[2021,10,26]]}},"alternative-id":["10.1073\/pnas.2103091118"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2103091118","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,20]]},"assertion":[{"value":"2021-08-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2103091118"}}