{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T15:06:51Z","timestamp":1775228811585,"version":"3.50.1"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2022,8,1]],"date-time":"2022-08-01T00:00:00Z","timestamp":1659312000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2022,8,1]],"date-time":"2022-08-01T00:00:00Z","timestamp":1659312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,9]]},"DOI":"10.1007\/s10994-022-06192-x","type":"journal-article","created":{"date-parts":[[2022,8,1]],"date-time":"2022-08-01T21:02:46Z","timestamp":1659387766000},"page":"3359-3392","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Stabilize deep ResNet with a sharp scaling factor $$\\tau$$"],"prefix":"10.1007","volume":"111","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2711-7295","authenticated-orcid":false,"given":"Huishuai","family":"Zhang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Da","family":"Yu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingyang","family":"Yi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tie-Yan","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,8,1]]},"reference":[{"key":"6192_CR1","unstructured":"Allen-Zhu, Z., & Li, Y. (2019). What can ResNet learn efficiently, going beyond kernels? Advances in Neural Information Processing Systems."},{"key":"6192_CR2","unstructured":"Allen-Zhu, Z., Li, Y., & Song, Z. (2018). A convergence theory for deep learning via over-parameterization. arXiv preprint arXiv:1811.03962."},{"key":"6192_CR3","unstructured":"Allen-Zhu, Z., Li, Y., & Liang, Y. (2019a). Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in Neural Information Processing Systems, pp.6155\u20136166."},{"key":"6192_CR4","unstructured":"Allen-Zhu, Z., Li, Y., & Song, Z. (2019b). On the convergence rate of training recurrent neural networks. Advances in Neural Information Processing Systems."},{"key":"6192_CR5","unstructured":"Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., & Wang, R. (2019a). On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems."},{"key":"6192_CR6","unstructured":"Arora, S., Du, S. S., Hu, W., Li, Z., & Wang, R. (2019b). Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Learning (ICML)."},{"key":"6192_CR7","unstructured":"Arpit, D., Campos, V., & Bengio, Y. (2019). How to initialize your network? robust initialization for weightnorm & resnets. Advances in Neural Information Processing Systems."},{"key":"6192_CR8","unstructured":"Balduzzi, D., Frean, M., Leary, L., Lewis, J. P., Wan-Duo Ma, K., & McWilliams, B. (2017). The shattered gradients problem: If resnets are the answer, then what is the question? In International Conference on Machine Learning (ICML), pp. 342\u2013350."},{"key":"6192_CR9","unstructured":"Brutzkus, A., Globerson, A., Malach, E., & Shalev-Shwartz, S. (2018). SGD learns over-parameterized networks that provably generalize on linearly separable data. In Proceedings of the 6th international conference on learning representations (ICLR 2018)."},{"key":"6192_CR10","doi-asserted-by":"crossref","unstructured":"Cao, Y., & Gu, Q. (2019). A generalization theory of gradient descent for learning over-parameterized deep ReLU networks. arXiv preprint arXiv:1902.01384.","DOI":"10.1609\/aaai.v34i04.5736"},{"key":"6192_CR11","unstructured":"Cao, Y., & Gu, Q. (2020). Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in Neural Information Processing Systems (NeurIPS)."},{"key":"6192_CR12","unstructured":"Chen, Z., Cao, Y., Zou, D., & Gu, Q. (2021). How much over-parameterization is sufficient to learn deep ReLU networks? In Proceedings of the international conference on learning representations (ICLR 2021)."},{"key":"6192_CR13","unstructured":"Chizat, L., & Bach, F. (2018a). On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in Neural Information Processing Systems 31."},{"key":"6192_CR14","unstructured":"Chizat, L., & Bach, F. (2018b). A note on lazy training in supervised differentiable programming. arXiv preprint arXiv:1812.07956, 8."},{"key":"6192_CR15","unstructured":"Chizat, L., Oyallon, E., & Bach, F. (2019). On lazy training in differentiable programming. Advances in Neural Information Processing Systems."},{"key":"6192_CR16","unstructured":"Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT."},{"key":"6192_CR17","unstructured":"Du, S. S., Lee, J. D., Li, H., Wang, L., & Zhai, X. (2019a). Gradient descent finds global minima of deep neural networks. In: International Conference on Machine Learning (ICML)."},{"key":"6192_CR18","unstructured":"Du, S. S., Zhai, X., Poczos, B., & Singh, A. (2019b). Gradient descent provably optimizes over-parameterized neural networks. In: International Conference on Learning Representations (ICLR)."},{"key":"6192_CR19","unstructured":"Fang, C., Dong, H., & Zhang, T. (2019a). Over parameterized two-level neural networks can learn near optimal feature representations. arXiv preprint arXiv:1910.11508."},{"key":"6192_CR20","unstructured":"Fang, C., Gu, Y., Zhang, W., & Zhang, T. (2019b). Convex formulation of overparameterized deep neural networks. arXiv preprint arXiv:1911.07626."},{"key":"6192_CR21","unstructured":"Frei, S., Cao, Y., & Gu, Q. (2019). Algorithm-dependent generalization bounds for overparameterized deep residual networks. Advances in Neural Information Processing Systems, pages 14769\u201314779."},{"key":"6192_CR22","unstructured":"Ghorbani, B., Mei, S., Misiakiewicz, T., Montanari, A. (2019). Limitations of lazy training of two-layers neural networks. Advances in Neural Information Processing Systems."},{"issue":"1","key":"6192_CR23","doi-asserted-by":"publisher","first-page":"014004","DOI":"10.1088\/1361-6420\/aa9a90","volume":"34","author":"E Haber","year":"2017","unstructured":"Haber, E., & Ruthotto, L. (2017). Stable architectures for deep neural networks. Inverse Problems, 34(1), 014004.","journal-title":"Inverse Problems"},{"key":"6192_CR24","unstructured":"Hardt, M., & Ma, T. (2016). Identity matters in deep learning. In: International Conference on Learning Representations (ICLR)."},{"key":"6192_CR25","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","DOI":"10.1109\/CVPR.2016.90"},{"key":"6192_CR26","unstructured":"Ioffe, S., & Szegedy, C. (2015). Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (ICML), pp. 448\u2013456."},{"key":"6192_CR27","unstructured":"Jacot, A., Gabriel, F, & Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, pp. 8571\u20138580."},{"key":"6192_CR28","unstructured":"Ji, Z., & Telgarsky, M. (2020). Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks. In Proceedings of the international conference on learning representations (ICLR 2020)."},{"key":"6192_CR29","unstructured":"Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images."},{"key":"6192_CR30","doi-asserted-by":"crossref","unstructured":"Laurent, B., & Massart, P. (2000). Adaptive estimation of a quadratic functional by model selection. Annals of Statistics, pp. 1302\u20131338.","DOI":"10.1214\/aos\/1015957395"},{"issue":"11","key":"6192_CR31","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","volume":"86","author":"Y LeCun","year":"1998","unstructured":"LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278\u20132324.","journal-title":"Proceedings of the IEEE"},{"key":"6192_CR32","unstructured":"Li, Y., & Liang, Y. (2018). Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in Neural Information Processing Systems, pp. 8168\u20138177."},{"issue":"33","key":"6192_CR33","doi-asserted-by":"publisher","first-page":"E7665","DOI":"10.1073\/pnas.1806579115","volume":"115","author":"S Mei","year":"2018","unstructured":"Mei, S., Montanari, A., & Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33), E7665\u2013E7671.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"6192_CR34","unstructured":"Mei, S., Misiakiewicz, T., & Montanari, A. (2019). Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Proceedings of the thirty-second conference on learning theory (pp. 2388\u20132464)."},{"key":"6192_CR35","unstructured":"Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., & Srebro, N. (2019). The role of over-parametrization in generalization of neural networks. In: International Conference on Learning Representations (ICLR)."},{"key":"6192_CR36","unstructured":"Nguyen. P.-M. (2019). Mean field limit of the learning dynamics of multilayer neural networks. arXiv preprint arXiv:1902.02880."},{"key":"6192_CR37","unstructured":"Orhan, A. E., & Pitkow, X. (2018). Skip connections eliminate singularities. In: International Conference on Learning Representations (ICLR)."},{"key":"6192_CR38","unstructured":"Oymak, S., & Soltanolkotabi, M. (2019). Overparameterized nonlinear learning: Gradient descent takes the shortest path? In: International Conference on Machine Learning (ICML)."},{"key":"6192_CR39","doi-asserted-by":"crossref","unstructured":"Spielman, D. A., & Teng, S-H. (2004). Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. Journal of the ACM (JACM), 51(3):385\u2013463.","DOI":"10.1145\/990308.990310"},{"key":"6192_CR40","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, \u0141., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems."},{"key":"6192_CR41","unstructured":"Veit, A., Wilber, M. J., & Belongie, S. (2016). Residual networks behave like ensembles of relatively shallow networks. Advances in Neural Information Processing Systems, pp 550\u2013558."},{"key":"6192_CR42","first-page":"210","volume-title":"Introduction to the non-asymptotic analysis of random matrices","author":"R Vershynin","year":"2012","unstructured":"Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices (pp. 210\u2013268). Theory and Applications: Compressed Sensing."},{"key":"6192_CR43","unstructured":"Yang, G., and Schoenholz, S. (2017). Mean field residual networks: On the edge of chaos. Advances in Neural Information Processing Systems, pp 7103\u20137114."},{"key":"6192_CR44","unstructured":"Yehudai, G., & Shamir, O. (2019). On the power and limitations of random features for understanding neural networks. Advances in Neural Information Processing Systems."},{"key":"6192_CR45","unstructured":"Zhang, H., Dauphin, Y. N., & Ma, T. (2019a). Fixup initialization: Residual learning without normalization. In: International Conference on Learning Representations (ICLR)."},{"key":"6192_CR46","unstructured":"Zhang, H., Chen, W., & Liu, T.-Y. (2018). On the local hessian in back-propagation. In Advances in Neural Information Processing Systems, pp. 6521\u20136531."},{"key":"6192_CR47","doi-asserted-by":"crossref","unstructured":"Zhang, J., Han, B., Wynter, L., Low, K. H., & Kankanhalli, M. (2019b). Towards robust resnet: A small step but a giant leap. In: International Joint Conferences on Artificial Intelligence (IJCAI).","DOI":"10.24963\/ijcai.2019\/595"},{"key":"6192_CR48","unstructured":"Zou, D., & Gu, Q. (2019). An improved analysis of training over-parameterized deep neural networks. Advances in Neural Information Processing Systems."},{"key":"6192_CR49","doi-asserted-by":"crossref","unstructured":"Zou, D., Cao, Y., Zhou, D., & Gu, Q. (2020). Stochastic gradient descent optimizes over-parameterized deep ReLU networks. Machine Learning, 109(3), 467\u2013492.","DOI":"10.1007\/s10994-019-05839-6"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06192-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-022-06192-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06192-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,1]],"date-time":"2023-08-01T00:03:33Z","timestamp":1690848213000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-022-06192-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,1]]},"references-count":49,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2022,9]]}},"alternative-id":["6192"],"URL":"https:\/\/doi.org\/10.1007\/s10994-022-06192-x","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,1]]},"assertion":[{"value":"26 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 May 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 August 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 August 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}