{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T19:37:38Z","timestamp":1790797058352,"version":"4.1.0"},"reference-count":45,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,2,7]],"date-time":"2020-02-07T00:00:00Z","timestamp":1581033600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["11472272"],"award-info":[{"award-number":["11472272"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Deep learning plays a key role in the recent developments of machine learning. This paper develops a deep residual neural network (ResNet) for the regression of nonlinear functions. Convolutional layers and pooling layers are replaced by fully connected layers in the residual block. To evaluate the new regression model, we train and test neural networks with different depths and widths on simulated data, and we find the optimal parameters. We perform multiple numerical tests of the optimal regression model on multiple simulated data, and the results show that the new regression model behaves well on simulated data. Comparisons are also made between the optimal residual regression and other linear as well as nonlinear approximation techniques, such as lasso regression, decision tree, and support vector machine. The optimal residual regression model has better approximation capacity compared to the other models. Finally, the residual regression is applied into the prediction of a relative humidity series in the real world. Our study indicates that the residual regression model is stable and applicable in practice.<\/jats:p>","DOI":"10.3390\/e22020193","type":"journal-article","created":{"date-parts":[[2020,2,7]],"date-time":"2020-02-07T11:50:28Z","timestamp":1581076228000},"page":"193","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":107,"title":["Deep Residual Learning for Nonlinear Regression"],"prefix":"10.3390","volume":"22","author":[{"given":"Dongwei","family":"Chen","sequence":"first","affiliation":[{"name":"School of Mathematical and Statistical Sciences, Clemson University, Clemson, SC 29641, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei","family":"Hu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Atmospheric Boundary Layer Physics and Atmospheric Chemistry, Institute of Atmospheric Physics, Chinese Academy of Sciences, Beijing 100029, China"},{"name":"College of Earth Science, University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guokui","family":"Nian","sequence":"additional","affiliation":[{"name":"College of Earth Science, University of Chinese Academy of Sciences, Beijing 100049, China"},{"name":"State Key Laboratory of Numerical Modeling for Atmospheric Sciences and Geophysical Fluid Dynamics, Institute of Atmospheric Physics, Chinese Academy of Sciences, Beijing 100029, China"},{"name":"Forecast Weather (Suzhou) Technology Co., Ltd., Suzhou 215000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tiantian","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Mathematical and Statistical Sciences, Clemson University, Clemson, SC 29641, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,7]]},"reference":[{"key":"ref_1","first-page":"96","article-title":"Multiple linear regression and genetic algorithm approaches to predict temporal scour depth near circular pier in non-cohesive sediment","volume":"26","author":"Pandey","year":"2020","journal-title":"Ish. J. Hydraul. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1051","DOI":"10.1109\/TPWRS.2011.2172004","article-title":"A stepwise regression method for estimating dominant electromechanical modes","volume":"27","author":"Zhou","year":"2011","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"8369","DOI":"10.1016\/j.eswa.2012.01.183","article-title":"Using ridge regression with genetic algorithm to enhance real estate appraisal forecasting","volume":"39","author":"Ahn","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"Zou","year":"2005","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bates, D.M., and Watts, D.G. (1988). Nonlinear Regression Analysis and Its Applications, Wiley. [1st ed.].","DOI":"10.1002\/9780470316757"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Rhinehart, R.R. (2016). Nonlinear Regression Modeling for Engineering Applications: Modeling, Model Validation, and Enabling Design of Experiments, Wiley. [1st ed.].","DOI":"10.1002\/9781118597972"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ritz, C., and Streibig, J.C. (2008). Nonlinear Regression with R, Springer. [1st ed.].","DOI":"10.1007\/978-0-387-09616-2"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Motulsky, H., and Christopoulos, A. (2004). Fitting Models to Biological Data Using Linear and Nonlinear Regression: A Practical Guide to Curve Fitting, Oxford University Press. [1st ed.].","DOI":"10.1093\/oso\/9780195171792.001.0001"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1088\/0951-7715\/26\/1\/201","article-title":"Physics constrained nonlinear regression models for time series","volume":"26","author":"Majda","year":"2012","journal-title":"Nonlinearity"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2265","DOI":"10.1016\/j.eswa.2009.07.046","article-title":"Application of fuzzy inference system and nonlinear regression models for predicting rock brittleness","volume":"37","author":"Yagiz","year":"2010","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"3015","DOI":"10.1016\/j.atmosenv.2010.05.009","article-title":"An enhanced PM2. 5 air quality forecast model based on nonlinear regression and back-trajectory concentrations","volume":"44","author":"Cobourn","year":"2010","journal-title":"Atmos. Environ."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"708","DOI":"10.1016\/j.jhazmat.2007.03.041","article-title":"Application of nonlinear regression analysis for ammonium exchange by natural (Bigadic) clinoptilolite","volume":"148.3","author":"Gunay","year":"2007","journal-title":"J. Hazard. Mater."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1217","DOI":"10.1214\/08-AOAS210","article-title":"Parameter estimation for computationally intensive nonlinear regression with an application to climate modeling","volume":"2","author":"Drignei","year":"2008","journal-title":"Ann. Appl. Stat."},{"key":"ref_15","unstructured":"Hipp, L.B. (2013). The Weierstrass Approximation Theorem. [Master\u2019s Thesis, University of South Carolina]."},{"key":"ref_16","unstructured":"Drucker, H., Burges, C.C., Kaufman, L., Smola, A.J., and Vapnik, V.N. (1996). Advances in Neural Information Processing Systems 9, MIT Press. [1st ed.]."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1023\/B:STCO.0000035301.49549.88","article-title":"A tutorial on support vector regression","volume":"14","author":"Smola","year":"2004","journal-title":"Stat. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1002\/widm.8","article-title":"Classification and regression trees","volume":"1","author":"Loh","year":"2011","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/BF02551274","article-title":"Approximation by superpositions of a sigmoidal function","volume":"2","author":"Cybenko","year":"1989","journal-title":"Math. Control. Signals. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/0893-6080(89)90020-8","article-title":"Multilayer feedforward networks are universal approximators","volume":"2","author":"Hornik","year":"1989","journal-title":"Neural Netw."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/0893-6080(89)90003-8","article-title":"On the approximate realization of continuous mappings by neural networks","volume":"2","author":"Funahashi","year":"1989","journal-title":"Neural Netw."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1016\/0893-6080(92)90012-8","article-title":"Kolmogorov\u2019s theorem and multilayer neural networks","volume":"5","year":"1992","journal-title":"Neural Netw."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"879","DOI":"10.1109\/TNN.2006.875977","article-title":"Universal approximation using incremental constructive feedforward networks with random hidden nodes","volume":"17","author":"Huang","year":"2006","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/TNN.2004.836233","article-title":"Smooth function approximation using neural networks","volume":"16","author":"Ferrari","year":"2005","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1109\/72.728353","article-title":"On the approximation of stochastic processes by approximate identity neural networks","volume":"9","author":"Turchetti","year":"1998","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1016\/S0893-6080(99)00017-9","article-title":"Artificial neural networks as approximators of stochastic processes","volume":"12","author":"Belli","year":"1999","journal-title":"Neural Netw."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1109\/TNN.2007.2000055","article-title":"Representation of nonlinear random transformations by non-Gaussian stochastic neural networks","volume":"19","author":"Turchetti","year":"2008","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_29","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). Imagenet classification with deep convolutional neural networks. Proceedings of the 26th Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_30","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_32","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the 29th IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA."},{"key":"ref_33","unstructured":"Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. (2017, January 4\u20139). The expressive power of neural networks: A view from the width. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_34","unstructured":"Hanin, B., and Sellke, M. (2017). Approximating continuous functions by ReLU nets of minimal width. Arxiv Prepr."},{"key":"ref_35","unstructured":"Lin, H., and Jegelka, S. (2018, January 2\u20138). ResNet with one-neuron hidden layers is a Universal Approximator. Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, Montr\u00e9al, QC, Canada."},{"key":"ref_36","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 7\u20139). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, Lille, France."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Hara, K., Saito, D., and Shouno, H. (2015, January 12\u201316). Analysis of function of rectified linear unit used in deep learning. Proceedings of the International Joint Conference on Neural Networks, Killarney, Ireland.","DOI":"10.1109\/IJCNN.2015.7280578"},{"key":"ref_38","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference for Learning Representations, San Diego, CA, USA."},{"key":"ref_39","first-page":"2121","article-title":"Adaptive subgradient methods for online learning and stochastic optimization","volume":"12","author":"Duchi","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_40","first-page":"26","article-title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","volume":"4","author":"Tieleman","year":"2012","journal-title":"COURSERA Neural Netw. Mach. Learn."},{"key":"ref_41","unstructured":"Ian, G., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press. [1st ed.]."},{"key":"ref_42","unstructured":"(2020, January 27). Homepage of Scikit-Learn. Available online: https:\/\/scikit-learn.org\/stable\/."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"D14204","DOI":"10.1029\/2006JD008136","article-title":"Effects of aerosols and relative humidity on cumulus clouds","volume":"112","author":"Fan","year":"2007","journal-title":"J. Geophys. Res. Atmos."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"10606","DOI":"10.1029\/2018JD028315","article-title":"Method to Estimate Water Vapor Supersaturation in the Ambient Activation Process Using Aerosol and Droplet Measurement Data","volume":"123","author":"Shen","year":"2018","journal-title":"J. Geophys. Res. Atmos."},{"key":"ref_45","unstructured":"(2020, January 27). Download Page of ERA5 Hourly Data on Pressure Levels from 1979 to Present. Available online: https:\/\/cds.climate.copernicus.eu\/cdsapp#!\/dataset\/10.24381\/cds.bd0915c6?tab=form."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/193\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:55:40Z","timestamp":1760172940000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/2\/193"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,7]]},"references-count":45,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["e22020193"],"URL":"https:\/\/doi.org\/10.3390\/e22020193","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,7]]}}}