{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T13:59:14Z","timestamp":1753883954889,"version":"3.41.2"},"reference-count":27,"publisher":"World Scientific Pub Co Pte Ltd","issue":"01","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["11871051"],"award-info":[{"award-number":["11871051"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Beijing Natural Science Foundation Project","award":["Z200002"],"award-info":[{"award-number":["Z200002"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Asia Pac. J. Oper. Res."],"published-print":{"date-parts":[[2023,2]]},"abstract":"<jats:p> Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factorization (EKFAC), proposed an interpretation by viewing natural gradient update as a diagonal method and corrects the inaccurate re-scaling factor in the KFAC eigenbasis. What\u2019s more, a new method to approximate the natural gradient called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) is also proposed, in which the Fisher information matrix (FIM) is approximated as a constant multiplied by the Kronecker product of two matrices and the traces can be kept equal before and after the approximation. In this work, we combine the ideas of these two methods and propose Trace-restricted Eigenvalue-corrected Kronecker Factorization (TEKFAC). The proposed method not only corrects the inexact re-scaling factor under the Kronecker-factored eigenbasis, but also considers the new approximation method and the effective damping technique adopted by TKFAC. We also discuss the differences and relationships among the related Kronecker-factored approximations. Empirically, our method outperforms SGD with momentum, Adam, EKFAC and TKFAC on several DNNs. <\/jats:p>","DOI":"10.1142\/s0217595923400055","type":"journal-article","created":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T00:52:05Z","timestamp":1673571125000},"source":"Crossref","is-referenced-by-count":0,"title":["Eigenvalue-Corrected Natural Gradient Based on a New Approximation"],"prefix":"10.1142","volume":"40","author":[{"given":"Kaixin","family":"Gao","sequence":"first","affiliation":[{"name":"School of Mathematics, Tianjin University, Tianjin 300350, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zheng-Hai","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Mathematics, Tianjin University, Tianjin 300350, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaolei","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Mathematics, Tianjin University, Tianjin 300350, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Min","family":"Wang","sequence":"additional","affiliation":[{"name":"Central Software Institute, Huawei Technologies Co. Ltd, Hangzhou 310051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuangling","family":"Wang","sequence":"additional","affiliation":[{"name":"Central Software Institute, Huawei Technologies Co. Ltd, Hangzhou 310051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zidong","family":"Wang","sequence":"additional","affiliation":[{"name":"Central Software Institute, Huawei Technologies Co. Ltd, Hangzhou 310051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dachuan","family":"Xu","sequence":"additional","affiliation":[{"name":"Beijing Institute for Scientific and Engineering Computing, Beijing University of Technology, Beijing 100124, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fan","family":"Yu","sequence":"additional","affiliation":[{"name":"Central Software Institute, Huawei Technologies Co. Ltd, Hangzhou 310051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2023,3,13]]},"reference":[{"issue":"2","key":"S0217595923400055BIB001","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1162\/089976698300017746","volume":"10","author":"Amari SI","year":"1998","journal-title":"Neural Computation"},{"volume-title":"Int. Conf. Learning Representations","year":"2017","author":"Ba J","key":"S0217595923400055BIB002"},{"volume-title":"Workshop of Bayesian Deep Learning, Advances in Neural Information Processing Systems","year":"2018","author":"Bae J","key":"S0217595923400055BIB003"},{"issue":"8","key":"S0217595923400055BIB005","doi-asserted-by":"crossref","first-page":"7046","DOI":"10.1609\/aaai.v35i8.16867","volume":"35","author":"Chen M","year":"2021","journal-title":"Proc. AAAI Conf. Artificial Intelligence"},{"issue":"1","key":"S0217595923400055BIB006","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1137\/1019005","volume":"19","author":"Dennis JE","year":"1977","journal-title":"SIAM Review"},{"issue":"61","key":"S0217595923400055BIB007","first-page":"2121","volume":"12","author":"Duchi J","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"S0217595923400055BIB008","first-page":"7519","volume-title":"Proc. AAAI Conf. Artificial Intelligence","author":"Gao K","year":"2021"},{"key":"S0217595923400055BIB009","first-page":"9550","volume-title":"Advances in Neural Information Processing Systems","author":"George T","year":"2018"},{"key":"S0217595923400055BIB010","first-page":"2386","volume-title":"Advances in Neural Information Processing Systems","author":"Goldfarb D","year":"2020"},{"key":"S0217595923400055BIB011","first-page":"573","volume-title":"Int. Conf. Machine Learning","author":"Grosse R","year":"2016"},{"key":"S0217595923400055BIB012","first-page":"770","volume-title":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","author":"He K","year":"2016"},{"volume-title":"Int. Conf. Learning Representations","year":"2014","author":"Kingma DP","key":"S0217595923400055BIB013"},{"volume-title":"Int. Conf. Learning Representations","year":"2013","author":"Kiros R","key":"S0217595923400055BIB014"},{"key":"S0217595923400055BIB016","first-page":"265","volume-title":"Int. Conf. Machine Learning","author":"Le QV","year":"2011"},{"key":"S0217595923400055BIB017","first-page":"735","volume-title":"Int. Conf. Machine Learning","author":"Martens J","year":"2010"},{"volume-title":"Int. Conf. Learning Representations","year":"2018","author":"Martens J","key":"S0217595923400055BIB018"},{"key":"S0217595923400055BIB019","first-page":"2408","volume-title":"Int. Conf. Machine Learning","author":"Martens J","year":"2015"},{"issue":"2","key":"S0217595923400055BIB020","first-page":"372","volume":"27","author":"Nesterov Y","year":"1983","journal-title":"Soviet Mathematics Doklady"},{"volume-title":"Workshop of Deep Learning and Unsupervised Feature Learning, Advances in Neural Information Processing Systems","year":"2011","author":"Netzer Y","key":"S0217595923400055BIB021"},{"key":"S0217595923400055BIB022","first-page":"12359","volume-title":"Proc. IEEE Conf. Computer Vision and Pattern Recognition","author":"Osawa K","year":"2019"},{"issue":"2","key":"S0217595923400055BIB023","doi-asserted-by":"crossref","first-page":"R49","DOI":"10.1190\/geo2015-0595.1","volume":"82","author":"Pan W","year":"2017","journal-title":"Geophysics"},{"issue":"1","key":"S0217595923400055BIB024","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1016\/S0893-6080(98)00116-6","volume":"12","author":"Qian N","year":"1999","journal-title":"Neural Networks"},{"issue":"3","key":"S0217595923400055BIB025","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1214\/aoms\/1177729586","volume":"22","author":"Robbins H","year":"1951","journal-title":"The Annals of Mathematical Statistics"},{"volume-title":"Int. Conf. Learning Representations","year":"2015","author":"Simonyan K","key":"S0217595923400055BIB026"},{"issue":"2","key":"S0217595923400055BIB027","first-page":"26","volume":"4","author":"Tieleman T","year":"2012","journal-title":"COURSERA: Neural Networks for Machine Learning"},{"key":"S0217595923400055BIB029","first-page":"5847","volume-title":"Int. Conf. Machine Learning","author":"Zhang G","year":"2018"},{"volume-title":"Int. Conf. Learning Representations","year":"2019","author":"Zhang G","key":"S0217595923400055BIB030"}],"container-title":["Asia-Pacific Journal of Operational Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0217595923400055","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T05:51:26Z","timestamp":1678945886000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0217595923400055"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2]]},"references-count":27,"journal-issue":{"issue":"01","published-print":{"date-parts":[[2023,2]]}},"alternative-id":["10.1142\/S0217595923400055"],"URL":"https:\/\/doi.org\/10.1142\/s0217595923400055","relation":{},"ISSN":["0217-5959","1793-7019"],"issn-type":[{"type":"print","value":"0217-5959"},{"type":"electronic","value":"1793-7019"}],"subject":[],"published":{"date-parts":[[2023,2]]},"article-number":"2340005"}}