{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T14:21:35Z","timestamp":1753885295096,"version":"3.41.2"},"reference-count":25,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2021,11,10]],"date-time":"2021-11-10T00:00:00Z","timestamp":1636502400000},"content-version":"vor","delay-in-days":313,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61971297"],"award-info":[{"award-number":["61971297"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Computational Intelligence and Neuroscience"],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:p>An adaptive clamping method (SGD\u2010MS) based on the radius of curvature is designed to alleviate the local optimal oscillation problem in deep neural network, which combines the radius of curvature of the objective function and the gradient descent of the optimizer. The radius of curvature is considered as the threshold to separate the momentum term or the future gradient moving average term adaptively. In addition, on this basis, we propose an accelerated version (SGD\u2010MA), which further improves the convergence speed by using the method of aggregated momentum. Experimental results on several datasets show that the proposed methods effectively alleviate the local optimal oscillation problem and greatly improve the convergence speed and accuracy. A novel parameter updating algorithm is also provided in this paper for deep neural network.<\/jats:p>","DOI":"10.1155\/2021\/9882068","type":"journal-article","created":{"date-parts":[[2021,11,11]],"date-time":"2021-11-11T05:50:09Z","timestamp":1636609809000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["An Adaptive Deep Learning Optimization Method Based on Radius of Curvature"],"prefix":"10.1155","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8357-0110","authenticated-orcid":false,"given":"Jiahui","family":"Zhang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0873-4514","authenticated-orcid":false,"given":"Xinhao","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6240-5925","authenticated-orcid":false,"given":"Ke","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenrui","family":"Wen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2021,11,10]]},"reference":[{"key":"e_1_2_9_1_2","doi-asserted-by":"crossref","unstructured":"SchroffF. KalenichenkoD. andPhilbinJ. FaceNet: a unified embedding for face recognition and clustering Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) June 2015 Boston MA USA.","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"e_1_2_9_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/tpami.2016.2577031"},{"key":"e_1_2_9_3_2","first-page":"9","article-title":"Online learning in neural networks","author":"Bottou L.","year":"1998","journal-title":"Chapter Online Learning and Stochastic Approximations"},{"key":"e_1_2_9_4_2","unstructured":"SutskeverI. MartensJ. DahlG. andHintonG. On the importance of initialization and momentum in deep learning Proceedings of the 30th International Conference on International Conference on Machine Learning-Volume 28 (ICML\u201913) June 2013 Atlanta GA USA."},{"key":"e_1_2_9_5_2","first-page":"257","article-title":"Adaptive subgradient methods for online learning and stochastic optimization","volume":"12","author":"Duchi J.","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_9_6_2","unstructured":"RuderS. An overview of gradient descent optimization algorithms 2016 https:\/\/arxiv.org\/abs\/1609.04747."},{"key":"e_1_2_9_7_2","article-title":"Adam: a method for stochastic optimization","author":"Kingma D. P.","year":"2014","journal-title":"Computer Science"},{"key":"e_1_2_9_8_2","unstructured":"WilsonA. C. RoelofsR. SternM. SrebroN. andRechtB. The marginal value of adaptive gradient methods in machine learning 2017 https:\/\/arxiv.org\/abs\/1705.08292."},{"key":"e_1_2_9_9_2","unstructured":"ImD. J. TaoM. andBransonK. An empirical analysis of the optimization of deep network loss surfaces 2016 https:\/\/arxiv.org\/abs\/1612.04010."},{"key":"e_1_2_9_10_2","unstructured":"ZhouZ. ZhangQ. LuG. WangH. ZhangW. andYuY. AdaShift: decorrelation and convergence of adaptive learning rate methods 2018 https:\/\/arxiv.org\/abs\/1810.00143."},{"key":"e_1_2_9_11_2","unstructured":"LiuL. JiangH. HeP. ChenW. LiuX. GaoJ. andHanJ. On the variance of the adaptive learning rate and beyond 2019 https:\/\/arxiv.org\/abs\/1908.03265."},{"volume-title":"Discrete-time Control Systems","year":"1995","author":"Ogata K.","key":"e_1_2_9_12_2"},{"key":"e_1_2_9_13_2","article-title":"A method of solving a convex programming problem with convergence rate O (1\/k2)","author":"Nesterov Y.","year":"1983","journal-title":"Soviet Mathematics Doklady"},{"key":"e_1_2_9_14_2","unstructured":"LucasJ. SunS. ZemelR. andGrosseR. Aggregated momentum: stability through passive damping 2019 https:\/\/arxiv.org\/abs\/1804.00325."},{"key":"e_1_2_9_15_2","doi-asserted-by":"crossref","unstructured":"AnW. WangH. SunQ. XuJ. DaiQ. andZhangL. A PID controller approach for stochastic optimization of deep networks Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) June 2018 Salt Lake City UT USA.","DOI":"10.1109\/CVPR.2018.00889"},{"key":"e_1_2_9_16_2","doi-asserted-by":"crossref","unstructured":"ShiL. ZhangY. WangW. ChengJ. andLuH. Rethinking the pid optimizer for stochastic optimization of deep networks Proceedings of the 2020 IEEE International Conference on Multimedia and Expo (ICME) July 2020 London UK 1\u20136.","DOI":"10.1109\/ICME46284.2020.9102970"},{"key":"e_1_2_9_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2963066"},{"key":"e_1_2_9_18_2","unstructured":"ZhangJ.andMitliagkasI. YellowFin and the art of momentum tuning 2017 https:\/\/arxiv.org\/abs\/1706.03471."},{"key":"e_1_2_9_19_2","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/3.3.175"},{"key":"e_1_2_9_20_2","unstructured":"GuptaV. KorenT. andSingerY. Shampoo: preconditioned stochastic tensor optimization 2018 https:\/\/arxiv.org\/abs\/1802.09568."},{"volume-title":"Systems of Extremal Control","year":"1974","author":"Rastrigin L. A.","key":"e_1_2_9_21_2"},{"key":"e_1_2_9_22_2","doi-asserted-by":"crossref","unstructured":"HeK. ZhangX. RenS. andSunJ. Deep residual learning for image recognition Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition December 2016 Las Vegas NV USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_9_23_2","doi-asserted-by":"crossref","unstructured":"HuangG. LiuZ. LaurensV. D. M. andWeinbergerK. Q. Densely connected convolutional networks Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) July 2017 Honolulu HI USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_2_9_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_9_25_2","article-title":"Learning multiple layers of features from tiny images","volume":"1","author":"Krizhevsky A.","year":"2009","journal-title":"Handbook of Systemic Autoimmune Diseases"}],"container-title":["Computational Intelligence and Neuroscience"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/cin\/2021\/9882068.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/cin\/2021\/9882068.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1155\/2021\/9882068","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T11:28:17Z","timestamp":1722943697000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1155\/2021\/9882068"}},"subtitle":[],"editor":[{"given":"Jos\u00e9 Alfredo","family":"Hern\u00e1ndez-P\u00e9rez","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,1]]},"references-count":25,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["10.1155\/2021\/9882068"],"URL":"https:\/\/doi.org\/10.1155\/2021\/9882068","archive":["Portico"],"relation":{},"ISSN":["1687-5265","1687-5273"],"issn-type":[{"type":"print","value":"1687-5265"},{"type":"electronic","value":"1687-5273"}],"subject":[],"published":{"date-parts":[[2021,1]]},"assertion":[{"value":"2021-06-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-19","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"9882068"}}