{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T18:27:50Z","timestamp":1775068070341,"version":"3.50.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,2,22]],"date-time":"2022-02-22T00:00:00Z","timestamp":1645488000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,22]],"date-time":"2022-02-22T00:00:00Z","timestamp":1645488000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:sec>\n                <jats:title>Abstract<\/jats:title>\n                <jats:p>Various deep neural network architectures (DNNs) maintain massive vital records in computer vision. While drawing attention worldwide, the design of the overall structure lacks general guidance. Based on the relationship between DNN design and numerical differential equations, we performed a fair comparison of the residual design with higher order perspectives. We show that the widely used DNN design strategy, constantly stacking a small design (usually, 2\u20133 layers), could be easily improved, supported by solid theoretical knowledge and with no extra parameters needed. We reorganise the residual design in higher order ways, which is inspired by the observation that many effective networks can be interpreted as different numerical discretisations of differential equations. The design of ResNet follows a relatively simple scheme, which is Euler forward; however, the situation becomes complicated rapidly while stacking. We suppose that stacked ResNet is somehow equalled to a higher order scheme; then, the current method of forwarding propagation might be relatively weak compared with a typical high-order method such as Runge\u2013Kutta. We propose HO-ResNet to verify the hypothesis on widely used CV benchmarks with sufficient experiments. Stable and noticeable increases in performance are observed, and convergence and robustness are also improved. Our stacking strategy improved ResNet-30 by 2.15% and ResNet-58 by 2.35% on CIFAR-10, with the same settings and parameters. The proposed strategy is fundamental and theoretical and can, therefore, be applied to any network as a general guideline.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Graphical abstract<\/jats:title>\n                \n              <\/jats:sec>","DOI":"10.1007\/s40747-022-00671-3","type":"journal-article","created":{"date-parts":[[2022,2,22]],"date-time":"2022-02-22T06:03:51Z","timestamp":1645509831000},"page":"3395-3407","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Rethinking ResNets: improved stacking strategies with high-order schemes for image classification"],"prefix":"10.1007","volume":"8","author":[{"given":"Zhengbo","family":"Luo","sequence":"first","affiliation":[]},{"given":"Zitang","family":"Sun","sequence":"additional","affiliation":[]},{"given":"Weilian","family":"Zhou","sequence":"additional","affiliation":[]},{"given":"Zizhang","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Sei-ichiro","family":"Kamata","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,2,22]]},"reference":[{"key":"671_CR1","unstructured":"Bachlechner T, Majumder BP, Mao HH, Cottrell G, McAuley J (2021) Rezero is all you need: fast convergence at large depth. In: de\u00a0Campos CP, Maathuis MH, Quaeghebeur E (eds.) Proceedings of the thirty-seventh conference on uncertainty in artificial intelligence, UAI 2021, Virtual Event, 27\u201330 July 2021, Proceedings of Machine Learning Research, vol. 161, pp. 1352\u20131361. AUAI Press. https:\/\/proceedings.mlr.press\/v161\/bachlechner21a.html"},{"key":"671_CR2","unstructured":"Bello I, Fedus W, Du X, Cubuk ED, Srinivas A, Lin TY, Shlens J, Zoph B (2021) Revisiting resnets: improved training and scaling strategies. arXiv preprint arXiv:2103.07579"},{"key":"671_CR3","doi-asserted-by":"publisher","unstructured":"Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Lechevallier Y, Saporta G (eds.) 19th international conference on computational statistics, COMPSTAT 2010, Paris, France, August 22\u201327, 2010\u2014Keynote, Invited and Contributed Papers. Physica-Verlag, pp 177\u2013186. https:\/\/doi.org\/10.1007\/978-3-7908-2604-3_16","DOI":"10.1007\/978-3-7908-2604-3_16"},{"issue":"1","key":"671_CR4","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1137\/050646032","volume":"29","author":"K Burrage","year":"2007","unstructured":"Burrage K, Lenane I, Lythe G (2007) Numerical methods for second-order stochastic differential equations. SIAM J Sci Comput 29(1):245\u2013264. https:\/\/doi.org\/10.1137\/050646032","journal-title":"SIAM J Sci Comput"},{"issue":"3","key":"671_CR5","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1007\/BF01932265","volume":"16","author":"JC Butcher","year":"1976","unstructured":"Butcher JC (1976) On the implementation of implicit Runge-Kutta methods. BIT Numer Math 16(3):237\u2013240","journal-title":"BIT Numer Math"},{"key":"671_CR6","unstructured":"Chen RT, Rubanova Y, Bettencourt J, Duvenaud DK (2018) Neural ordinary differential equations. In: Advances in neural information processing systems, pp 6571\u20136583"},{"key":"671_CR7","doi-asserted-by":"publisher","unstructured":"Cubuk ED, Zoph B, Man\u00e9 D, Vasudevan V, Le QV (2019) Autoaugment: learning augmentation strategies from data. In: IEEE conference on computer vision and pattern recognition, CVPR 2019, Long Beach, CA, USA, June 16\u201320, 2019, pp 113\u2013123. Computer Vision Foundation\/IEEE (2019). https:\/\/doi.org\/10.1109\/CVPR.2019.00020","DOI":"10.1109\/CVPR.2019.00020"},{"key":"671_CR8","unstructured":"De S, Smith SL (2020) Batch normalization biases residual blocks towards the identity function in deep networks. In: Larochelle H,\u00a0Ranzato M,\u00a0Hadsell R,\u00a0Balcan M,\u00a0Lin H (eds.) Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6\u201312, 2020, virtual"},{"key":"671_CR9","doi-asserted-by":"crossref","unstructured":"Ding X, Zhang X, Ma N, Han J, Ding G, Sun J (2021) Repvgg: making vgg-style convnets great again. In: IEEE conference on computer vision and pattern recognition, CVPR 2021, virtual, June 19\u201325, 2021, pp 13733\u201313742. Computer Vision Foundation\/IEEE","DOI":"10.1109\/CVPR46437.2021.01352"},{"key":"671_CR10","unstructured":"Gomez AN, Ren M, Urtasun R, Grosse RB (2017) The reversible residual network: backpropagation without storing activations. In:\u00a0Guyon I,\u00a0von Luxburg U, Bengio S, Wallach HM,\u00a0Fergus R, Vishwanathan SVN,\u00a0Garnett R (eds) Advances in neural information processing systems 30: annual conference on neural information processing systems 2017, December 4\u20139, 2017, Long Beach, CA, USA, pp 2214\u20132224"},{"key":"671_CR11","doi-asserted-by":"publisher","unstructured":"He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: 2015 IEEE international conference on computer vision, ICCV 2015, Santiago, Chile, December 7\u201313, 2015, pp 1026\u20131034. IEEE Computer Society. https:\/\/doi.org\/10.1109\/ICCV.2015.123","DOI":"10.1109\/ICCV.2015.123"},{"key":"671_CR12","doi-asserted-by":"publisher","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27\u201330, 2016, pp 770\u2013778. IEEE Computer Society. https:\/\/doi.org\/10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"key":"671_CR13","doi-asserted-by":"publisher","unstructured":"Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: 2018 IEEE conference on computer vision and pattern recognition, CVPR 2018, Salt Lake City, UT, USA, June 18\u201322, 2018, pp 7132\u20137141. Computer Vision Foundation\/IEEE Computer Society.https:\/\/doi.org\/10.1109\/CVPR.2018.00745. http:\/\/openaccess.thecvf.com\/content_cvpr_2018\/html\/Hu_Squeeze-and-Excitation_Networks_CVPR_2018_paper.html","DOI":"10.1109\/CVPR.2018.00745"},{"key":"671_CR14","doi-asserted-by":"publisher","unstructured":"Huang G, Liu Z, van\u00a0der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21\u201326, 2017, pp 2261\u20132269. IEEE Computer Society. https:\/\/doi.org\/10.1109\/CVPR.2017.243","DOI":"10.1109\/CVPR.2017.243"},{"key":"671_CR15","unstructured":"Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: F.R. Bach, D.M. Blei (eds.) Proceedings of the 32nd international conference on machine learning, ICML 2015, Lille, France, 6\u201311 July 2015, JMLR workshop and conference proceedings, vol.\u00a037, pp 448\u2013456. JMLR.org. http:\/\/proceedings.mlr.press\/v37\/ioffe15.html"},{"issue":"1","key":"671_CR16","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1214\/aop\/1022855419","volume":"26","author":"J Jacod","year":"1998","unstructured":"Jacod J, Protter P et al (1998) Asymptotic error distributions for the Euler method for stochastic differential equations. Ann Probab 26(1):267\u2013307","journal-title":"Ann Probab"},{"key":"671_CR17","unstructured":"Krizhevsky A, Hinton G, et\u00a0al (2009) Learning multiple layers of features from tiny images"},{"key":"671_CR18","doi-asserted-by":"publisher","unstructured":"Lee Y, Hwang J, Lee S, Bae Y, Park J (2019) An energy and gpu-computation efficient backbone network for real-time object detection. In: IEEE conference on computer vision and pattern recognition workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16\u201320, 2019, pp. 752\u2013760. Computer Vision Foundation\/IEEE. https:\/\/doi.org\/10.1109\/CVPRW.2019.00103","DOI":"10.1109\/CVPRW.2019.00103"},{"key":"671_CR19","unstructured":"Li Y, Yao T, Pan Y, Mei T (2021) Contextual transformer networks for visual recognition. CoRR arxiv:2107.12292"},{"key":"671_CR20","unstructured":"Liao Q, Poggio TA (2016) Bridging the gaps between residual learning, recurrent neural networks and visual cortex. CoRR arXiv:1604.03640"},{"key":"671_CR21","unstructured":"Liu W, Rabinovich A, Berg AC (2015) Parsenet: looking wider to see better. CoRR arXiv:1506.04579"},{"key":"671_CR22","unstructured":"Lu Y, Zhong A, Li Q, Dong B (2018) Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In: Dy JG,\u00a0Krause A (eds) Proceedings of the 35th international conference on machine learning, ICML 2018, Stockholmsm\u00e4ssan, Stockholm, Sweden, July 10\u201315, 2018, Proceedings of machine learning research, vol.\u00a080, pp 3282\u20133291. PMLR. http:\/\/proceedings.mlr.press\/v80\/lu18d.html"},{"issue":"3","key":"671_CR23","doi-asserted-by":"publisher","first-page":"352","DOI":"10.1007\/s10851-019-00903-1","volume":"62","author":"L Ruthotto","year":"2020","unstructured":"Ruthotto L, Haber E (2020) Deep neural networks motivated by partial differential equations. J Math Imaging Vis 62(3):352\u2013364. https:\/\/doi.org\/10.1007\/s10851-019-00903-1","journal-title":"J Math Imaging Vis"},{"issue":"173","key":"671_CR24","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1090\/S0025-5718-1986-0815836-3","volume":"46","author":"LF Shampine","year":"1986","unstructured":"Shampine LF (1986) Some practical Runge-Kutta formulas. Math Comput 46(173):135\u2013150","journal-title":"Math Comput"},{"key":"671_CR25","unstructured":"Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: Bengio Y, LeCunY (eds.) 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, conference track proceedings. arXiv:1409.1556"},{"key":"671_CR26","unstructured":"Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. CoRR arXiv:1505.00387"},{"key":"671_CR27","unstructured":"Tan M, Le QV (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th international conference on machine learning, ICML 2019, 9\u201315 June 2019, Long Beach, California, USA, Proceedings of machine learning research, vol.\u00a097, pp 6105\u20136114. PMLR. http:\/\/proceedings.mlr.press\/v97\/tan19a.html"},{"key":"671_CR28","doi-asserted-by":"crossref","unstructured":"Touvron H, Cord M, Sablayrolles A, Synnaeve G, J\u00e9gou H (2021) Going deeper with image transformers. arXiv preprint arXiv:2103.17239","DOI":"10.1109\/ICCV48922.2021.00010"},{"issue":"4","key":"671_CR29","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1137\/0715051","volume":"15","author":"JH Verner","year":"1978","unstructured":"Verner JH (1978) Explicit Runge-Kutta methods with estimates of the local truncation error. SIAM J Numer Anal 15(4):772\u2013790","journal-title":"SIAM J Numer Anal"},{"issue":"1","key":"671_CR30","first-page":"1","volume":"5","author":"E Weinan","year":"2017","unstructured":"Weinan E (2017) A proposal on machine learning via dynamical systems. Commun Math Stat 5(1):1\u201311","journal-title":"Commun Math Stat"},{"key":"671_CR31","unstructured":"Zhang H, Dauphin YN, Ma T (2019) Fixup initialization: residual learning without normalization. In: 7th international conference on learning representations, ICLR 2019, New Orleans, LA, USA, May 6\u20139, 2019. OpenReview.net. https:\/\/openreview.net\/forum?id=H1gsz30cKX"},{"key":"671_CR32","doi-asserted-by":"publisher","unstructured":"Zhang X, Li Z, Loy CC, Lin D (2017) Polynet: a pursuit of structural diversity in very deep networks. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, July 21\u201326, 2017, pp 3900\u20133908. IEEE Computer Society. https:\/\/doi.org\/10.1109\/CVPR.2017.415","DOI":"10.1109\/CVPR.2017.415"},{"key":"671_CR33","unstructured":"Zhu M, Fu C (2018) Convolutional neural networks combined with runge-kutta methods. CoRR arXiv:1802.08831"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00671-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-022-00671-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00671-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,3]],"date-time":"2022-08-03T10:32:03Z","timestamp":1659522723000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-022-00671-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,22]]},"references-count":33,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["671"],"URL":"https:\/\/doi.org\/10.1007\/s40747-022-00671-3","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,22]]},"assertion":[{"value":"27 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 February 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}