{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T06:36:22Z","timestamp":1790836582649,"version":"4.1.0"},"reference-count":78,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"3","funder":[{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation","doi-asserted-by":"crossref","award":["197041"],"award-info":[{"award-number":["197041"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Platform for Advanced Scientific Computing","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Sci. Comput."],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:p>We propose a globally convergent multilevel training method for deep residual networks (ResNets). The devised method can be seen as a novel variant of the recursive multilevel trustregion (RMTR) method, which operates in hybrid (stochastic-deterministic) settings by adaptively adjusting minibatch sizes during the training. The multilevel hierarchy and the transfer operators are constructed by exploiting a dynamical system's viewpoint, which interprets forward propagation through the ResNet as a forward Euler discretization of an initial value problem. In contrast to traditional training approaches, our novel RMTR method also incorporates curvature information on all levels of the multilevel hierarchy by means of the limited-memory SR1 method. The overall performance and the convergence properties of the our multilevel training method are numerically investigated using examples from the field of classification and regression.<\/jats:p>","DOI":"10.1137\/21m1434076","type":"journal-article","created":{"date-parts":[[2022,8,30]],"date-time":"2022-08-30T11:57:14Z","timestamp":1661860634000},"page":"S254-S280","source":"Crossref","is-referenced-by-count":11,"title":["Globally Convergent Multilevel Training of Deep Residual Networks"],"prefix":"10.1137","volume":"45","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8388-5518","authenticated-orcid":true,"given":"Alena","family":"Kopani\u010d\u00e1kov\u00e1","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rolf","family":"Krause","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2022,8,30]]},"reference":[{"key":"atypb1","doi-asserted-by":"publisher","DOI":"10.1007\/s00211-018-0977-z"},{"key":"atypb2","volume-title":"Just Sample, preprint, arXiv:1901.09997","author":"Berahas A. S.","year":"2019"},{"key":"atypb3","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2019.1658107"},{"key":"atypb4","doi-asserted-by":"publisher","DOI":"10.1287\/ijoo.2019.0016"},{"key":"atypb5","doi-asserted-by":"publisher","DOI":"10.1137\/17M1154679"},{"key":"atypb6","doi-asserted-by":"publisher","DOI":"10.1137\/16M1080173"},{"key":"atypb7","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1977-0431719-X"},{"key":"atypb8","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719505"},{"key":"atypb9","doi-asserted-by":"publisher","DOI":"10.1007\/s10589-016-9868-3"},{"key":"atypb10","volume-title":"On the Approximation of the Solution of Partial Differential Equations by Artificial Neural Networks Trained by a Multilevel Levenberg-Marquardt Method, preprint, arXiv:1904.04685","author":"Calandra H.","year":"2019"},{"key":"atypb11","doi-asserted-by":"publisher","DOI":"10.1137\/19M1255355"},{"key":"atypb12","volume-title":"Multi-Level Residual Networks from Dynamical Systems View, preprint, arXiv:1710.10348","author":"Chang B.","year":"2017"},{"key":"atypb13","volume-title":"Backpropagation: Theory, Architectures, and Applications","author":"Chauvin Y.","year":"1995"},{"key":"atypb14","doi-asserted-by":"publisher","DOI":"10.1093\/europace\/euaa402"},{"key":"atypb15","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2017.2699184"},{"key":"atypb16","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-017-1141-8"},{"key":"atypb17","volume-title":"Keras","author":"Chollet F.","year":"2015"},{"key":"atypb18","first-page":"709","volume":"34","author":"Clarke F.","year":"2005","journal-title":"Control Cybernet."},{"key":"atypb19","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719857"},{"key":"atypb20","volume-title":"Multilevel Initialization for Layer-Parallel Deep Neural Network Training, preprint, arXiv:1912.08974","author":"Cyr E. C.","year":"2019"},{"key":"atypb21","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2019.1624747"},{"key":"atypb22","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2019.1624747"},{"key":"atypb23","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.194"},{"key":"atypb24","doi-asserted-by":"publisher","DOI":"10.1051\/proc\/202171131"},{"key":"atypb25","doi-asserted-by":"publisher","DOI":"10.1016\/j.nucengdes.2009.05.021"},{"key":"atypb26","volume-title":"On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent, preprint, arXiv:1811.12941","author":"Golmant N.","year":"2018"},{"key":"atypb27","volume-title":"Deep Learning","author":"Goodfellow I.","year":"2016"},{"key":"atypb28","doi-asserted-by":"publisher","DOI":"10.1093\/imanum\/drn034"},{"key":"atypb29","doi-asserted-by":"publisher","DOI":"10.1093\/imanum\/drx043"},{"key":"atypb30","doi-asserted-by":"publisher","DOI":"10.1137\/050623012"},{"key":"atypb31","doi-asserted-by":"publisher","DOI":"10.1137\/08071819X"},{"key":"atypb32","volume-title":"Mollifying Networks, preprint, arXiv:1608.04980","author":"Gulcehre C.","year":"2016"},{"key":"atypb33","volume-title":"Spline Parameterization of Neural Network Controls for Deep Learning, preprint, arXiv:2103.00301","author":"G\u00fcnther S.","year":"2021"},{"key":"atypb34","volume-title":"Layer-Parallel Training of Deep Residual Neural Networks, preprint, arXiv:1812.04352","author":"G\u00fcnther S.","year":"2018"},{"key":"atypb35","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6420\/aa9a90"},{"key":"atypb36","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","author":"Haber E.","year":"2018"},{"key":"atypb37","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-02427-0"},{"key":"atypb38","doi-asserted-by":"publisher","DOI":"10.1007\/BF01272517"},{"key":"atypb39","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"atypb40","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_38"},{"key":"atypb41","volume-title":"Dynamical Systems, and Linear Algebra, Pure Appl. Math. 60","author":"Hirsch M.","year":"1974"},{"key":"atypb42","first-page":"1731","volume-title":"Advances in Neural Information Processing Systems, M. I. Jordan, Y. LeCun, and S","author":"Hoffer E.","year":"2017"},{"key":"atypb43","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46493-0_39"},{"key":"atypb44","volume-title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, preprint, arXiv:1502.03167","author":"Ioffe S.","year":"2015"},{"key":"atypb45","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.129"},{"key":"atypb46","volume-title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, preprint, arXiv:1609.04836","author":"Keskar N. S.","year":"2016"},{"key":"atypb47","volume-title":"Improving Generalization Performance by Switching from Adam to SGD, preprint, arXiv:1712.07628","author":"Keskar N. S.","year":"2017"},{"key":"atypb48","volume-title":"ArXiv","author":"Kingma D. P.","year":"2014"},{"key":"atypb49","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC43674.2020.9286180"},{"key":"atypb50","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2019.112720"},{"key":"atypb51","volume-title":"Multilevel Active-Set Trust-Region (MASTR) Method for Bound Constrained Minimization, preprint, arXiv:2103.14552","author":"Kopani\u010d\u00e1kov\u00e1 A.","year":"2021"},{"key":"atypb52","doi-asserted-by":"publisher","DOI":"10.1137\/18M1194870"},{"key":"atypb53","volume-title":"Adaptive Monotone Multigrid Methods for Nonlinear Variational Problems","author":"Kornhuber R.","year":"1997"},{"key":"atypb54","doi-asserted-by":"publisher","DOI":"10.1007\/s007910100052"},{"key":"atypb55","unstructured":"B. Kragel,\n                      Streamline diffusion POD models in optimization\n                      , PhD thesis, 2005."},{"key":"atypb56","volume-title":"Learning multiple layers of features from tiny images, tech. report TR-2009","author":"Krizhevsky A.","year":"2009"},{"key":"atypb57","volume-title":"Decoupled Weight Decay Regularization, preprint, arXiv:1711.05101","author":"Loshchilov I.","year":"2017"},{"key":"atypb58","volume-title":"Dissecting Neural Odes, preprint, arXiv:2002.08071","author":"Massaroli S.","year":"2020"},{"key":"atypb59","volume-title":"An Adaptive Sample Size Trust-Region Method for Finite-Sum Minimization, preprint, arXiv:1910.03294","author":"Mohr R.","year":"2019"},{"key":"atypb60","doi-asserted-by":"publisher","DOI":"10.1080\/10556780008805795"},{"key":"atypb61","doi-asserted-by":"publisher","DOI":"10.1080\/10556788.2012.759571"},{"key":"atypb62","doi-asserted-by":"publisher","DOI":"10.1090\/S0025-5718-1980-0572855-7"},{"key":"atypb63","volume-title":"Numerical Optimization","author":"Nocedal J.","year":"2006"},{"key":"atypb64","volume-title":"Numer. Linear Algebra Appl.","author":"Ponce C.","year":"2022"},{"key":"atypb65","doi-asserted-by":"publisher","DOI":"10.1002\/nme.6077"},{"key":"atypb66","volume-title":"Continuous-In-Depth Neural Networks, preprint, arXiv:2008.02389","author":"Queiruga A. F.","year":"2020"},{"key":"atypb67","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2018.00081"},{"key":"atypb68","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729586"},{"key":"atypb69","volume-title":"Measuring the Effects of Data Parallelism on Neural Network Training, preprint, arXiv:1811.03600","author":"Shallue C. J.","year":"2018"},{"key":"atypb70","doi-asserted-by":"publisher","DOI":"10.4171\/PM\/1992"},{"key":"atypb71","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2011.37"},{"key":"atypb72","doi-asserted-by":"publisher","DOI":"10.1007\/s40304-017-0103-z"},{"key":"atypb73","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00023"},{"key":"atypb74","volume-title":"Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms, preprint, arXiv:1708.07747","author":"Xiao H.","year":"2017"},{"key":"atypb75","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-019-01405-z"},{"key":"atypb76","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611976236.23"},{"key":"atypb77","doi-asserted-by":"publisher","DOI":"10.1137\/080743160"},{"key":"atypb78","doi-asserted-by":"publisher","DOI":"10.1007\/s42514-021-00069-6"}],"container-title":["SIAM Journal on Scientific Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/21M1434076","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T17:53:15Z","timestamp":1787334795000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/21M1434076"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,30]]},"references-count":78,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1137\/21M1434076"],"URL":"https:\/\/doi.org\/10.1137\/21m1434076","relation":{},"ISSN":["1064-8275","1095-7197"],"issn-type":[{"value":"1064-8275","type":"print"},{"value":"1095-7197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,30]]}}}