{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,8,7]],"date-time":"2023-08-07T16:10:13Z","timestamp":1691424613192},"reference-count":24,"publisher":"MIT Press","issue":"9","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>This letter first constructs a typical solution of ResNets for multicategory classifications based on the idea of the gate control of LSTMs, from which a general interpretation of the ResNet architecture is given and the performance mechanism is explained. We also use more solutions to further demonstrate the generality of that interpretation. The classification result is then extended to the universal-approximation capability of the type of ResNet with two-layer gate networks, an architecture that was proposed in an original paper of ResNets and has both theoretical and practical significance.<\/jats:p>","DOI":"10.1162\/neco_a_01600","type":"journal-article","created":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T19:41:29Z","timestamp":1689190889000},"page":"1566-1592","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["On an Interpretation of ResNets via Gate-Network Control"],"prefix":"10.1162","volume":"35","author":[{"given":"Changcun","family":"Huang","sequence":"first","affiliation":[{"name":"Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C. cchuang@mail.ustc.edu.cn"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,8,7]]},"reference":[{"key":"2023080715480109200_bib1","author":"Aizawa","year":"2020","journal-title":"Universal approximation properties for ODENet and ResNet"},{"key":"2023080715480109200_bib2","article-title":"Multi-level residual networks from dynamical systems view","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Chang","year":"2018"},{"key":"2023080715480109200_bib3","article-title":"Neural ordinary differential equations","volume-title":"Advances in neural information processing systems","author":"Chen","year":"2018"},{"key":"2023080715480109200_bib4","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1017\/S0962492921000052","article-title":"Neural network approximation","volume":"30","author":"DeVore","year":"2021","journal-title":"Acta Numerica"},{"issue":"5","key":"2023080715480109200_bib5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s40304-017-0103-z","article-title":"A proposal on machine learning via dynamical systems","volume":"1","author":"E","year":"2017","journal-title":"Communications in Mathematics and Statistics"},{"issue":"1","key":"2023080715480109200_bib6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s40687-018-0172-y","article-title":"A mean-field optimal control formulation of deep learning","volume":"6","author":"E","year":"2019","journal-title":"Research in the Mathematical Sciences"},{"key":"2023080715480109200_bib7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-1-4613-0019-9","volume-title":"Convex polytopes","author":"Gr\u00fcbaum","year":"2003","edition":"2nd"},{"key":"2023080715480109200_bib8","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2023080715480109200_bib9","first-page":"630","article-title":"Identity mappings in deep residual networks","volume-title":"Proceedings of the European Conference on Computer Vision","author":"He","year":"2016"},{"issue":"8","key":"2023080715480109200_bib10","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"issue":"5","key":"2023080715480109200_bib11","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/0893-6080(89)90020-8","article-title":"Multilayer feedforward networks are universal approximators","volume":"2","author":"Hornik","year":"1989","journal-title":"Neural Networks"},{"issue":"11","key":"2023080715480109200_bib12","doi-asserted-by":"crossref","first-page":"2249","DOI":"10.1162\/neco_a_01316","article-title":"ReLU networks are universal approximators via piecewise linear or constant functions","volume":"32","author":"Huang","year":"2020","journal-title":"Neural Computation"},{"issue":"5","key":"2023080715480109200_bib13","doi-asserted-by":"crossref","first-page":"1671","DOI":"10.4171\/JEMS\/1221","article-title":"Deep learning via dynamical systems: An approximation perspective","volume":"25","author":"Li","year":"2023","journal-title":"Journal of the European Mathematical Society"},{"key":"2023080715480109200_bib14","article-title":"Resnet with one-neuron hidden layers is a universal approximator","volume-title":"Advances in neural information processing systems","author":"Lin","year":"2018","edition":"(Eds.)"},{"key":"2023080715480109200_bib15","first-page":"4922","article-title":"Approximation and non-parametric estimation of ResNet-type convolutional neural networks","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Oono","year":"2019"},{"key":"2023080715480109200_bib16","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1016\/j.jcp.2019.06.042","article-title":"Data driven governing equations approximation using deep neural networks","volume":"395","author":"Qin","year":"2019","journal-title":"Journal of Computational Physics"},{"issue":"6088","key":"2023080715480109200_bib17","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"2023080715480109200_bib18","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/3-540-49430-8_11","article-title":"Centering neural network gradient factors","volume-title":"Neural net- works: Tricks of the trade","author":"Schraudolph","year":"1998"},{"key":"2023080715480109200_bib19","author":"Srivastava","year":"2015","journal-title":"Highway networks"},{"issue":"5","key":"2023080715480109200_bib20","doi-asserted-by":"crossref","first-page":"2715","DOI":"10.1109\/TAC.2022.3190051","article-title":"Universal approximation power of deep residual neural networks through the lens of control","volume":"68","author":"Tabuada","year":"2023","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2023080715480109200_bib21","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/3-540-49430-8_10","article-title":"Solving the ill-conditioning in neural network learning","volume-title":"Neural networks: Tricks of the trade","author":"Van Der Smagt","year":"1998"},{"key":"2023080715480109200_bib22","article-title":"Attention is all you need","volume-title":"Advances in neural information processing systems","author":"Vaswani","year":"2017","edition":"(Eds.)"},{"key":"2023080715480109200_bib23","article-title":"Understanding deep learning requires rethinking generalization","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Zhang","year":"2017"},{"key":"2023080715480109200_bib24","first-page":"11086","article-title":"Approximation capabilities of neural ODEs and invertible residual networks","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zhang","year":"2020"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/35\/9\/1566\/2152744\/neco_a_01600.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/35\/9\/1566\/2152744\/neco_a_01600.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,7]],"date-time":"2023-08-07T15:48:39Z","timestamp":1691423319000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/35\/9\/1566\/116703\/On-an-Interpretation-of-ResNets-via-Gate-Network"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,7]]},"references-count":24,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,8,7]]},"published-print":{"date-parts":[[2023,8,7]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01600","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,9]]},"published":{"date-parts":[[2023,8,7]]}}}