{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:28:28Z","timestamp":1792787308773,"version":"4.5.0"},"reference-count":55,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"4","funder":[{"DOI":"10.13039\/100000181","name":"AFOSR","doi-asserted-by":"publisher","award":["MURI FA 9550 18-1-0502"],"award-info":[{"award-number":["MURI FA 9550 18-1-0502"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008208","name":"Courant Institute of Mathematical Sciences, New York University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008208","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Walter Haefner Foundation"},{"DOI":"10.13039\/501100012652","name":"ETH Z\u00fcrich Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100012652","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Sci. Comput."],"published-print":{"date-parts":[[2023,8,31]]},"abstract":"<jats:p>Abstract.<\/jats:p>\n                  <jats:p>We propose efficient numerical schemes for implementing the natural gradient descent (NGD) for a broad range of metric spaces with applications to PDE-based optimization problems. Our technique represents the natural gradient direction as a solution to a standard least-squares problem. Hence, instead of calculating, storing, or inverting the information matrix directly, we apply efficient methods from numerical linear algebra. We treat both scenarios where the Jacobian, i.e., the derivative of the state variable with respect to the parameter, is either explicitly known or implicitly given through constraints. We can thus reliably compute several natural NGDs for a large-scale parameter space. In particular, we are able to compute Wasserstein NGD in thousands of dimensions, which was believed to be out of reach. Finally, our numerical results shed light on the qualitative differences between the standard gradient descent and various NGD methods based on different metric spaces in nonconvex optimization problems.<\/jats:p>","DOI":"10.1137\/22m1477805","type":"journal-article","created":{"date-parts":[[2023,7,10]],"date-time":"2023-07-10T10:33:01Z","timestamp":1688985181000},"page":"A1621-A1655","source":"Crossref","is-referenced-by-count":23,"title":["Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems"],"prefix":"10.1137","volume":"45","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6227-0941","authenticated-orcid":true,"given":"Levon","family":"Nurbekyan","sequence":"first","affiliation":[{"name":"Department of Mathematics, University of California Los Angeles, Los Angeles, CA 90095 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wanzhou","family":"Lei","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, MA 02138 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7238-7978","authenticated-orcid":true,"given":"Yunan","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Cornell University, Ithaca, NY 14853 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2023,7,10]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-5056-2"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017746"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1109\/5.720251"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72162-6"},{"key":"ref5","series-title":"Lectures in Mathematics ETH Z\u00fcrich","volume-title":"Gradient Flows in Metric Spaces and in the Space of Probability Measures","author":"Ambrosio L.","year":"2008","edition":"2"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898719604"},{"key":"ref7","unstructured":"M. Arbel , \nA. Gretton , \nW. Li , and \nG. Montufar , Kernelized Wasserstein natural gradient, in International Conference on Learning Representations, 2020."},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1007\/s41884-020-00028-0"},{"key":"ref9","first-page":"1","volume":"38","author":"Davis T. A.","year":"2011","journal-title":"ACM Trans. Math. Softw. (TOMS)"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.4310\/MAA.2019.v26.n2.a3"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1002\/cpa.21990"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1093\/imanum\/drz034"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1002\/cpa.22069"},{"key":"ref14","series-title":"Johns Hopkins Studies in the Mathematical Sciences","volume-title":"Matrix Computations","author":"Golub G. H.","year":"1996","edition":"3"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1137\/090771806"},{"key":"ref16","unstructured":"N. D. Heavner , Building Rank-Revealing Factorizations with Randomization, Ph.D. thesis, University of Colorado at Boulder, 2019."},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1080\/03610919008812866"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1051\/cocv\/2021029"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1007\/s00211-020-01154-8"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1137\/18M118640X"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-26980-7_73"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-26980-7_74"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1007\/s41884-018-0015-3"},{"key":"ref24","unstructured":"W. Li  and \nJ. Zhao , Wasserstein Information Matrix, preprint, arXiv:1910.11248, 2019."},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-80209-7_57"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1016\/j.jde.2012.04.004"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1137\/20M1334668"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1137\/20M1344986"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.3934\/jdg.2021014"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-26980-7_62"},{"key":"ref31","first-page":"1","volume":"21","author":"Martens J.","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref32","unstructured":"J. Martens  and \nR. Grosse , Optimizing neural networks with Kronecker-factored approximate curvature, in International Conference on Machine Learning, PMLR, 2015, pp. 2408\u20132417."},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35289-8_27"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1137\/120877854"},{"key":"ref35","doi-asserted-by":"crossref","unstructured":"R. A. Meyer , \nC. Musco , \nC. Musco , and \nD. P. Woodruff , Hutch++: Optimal stochastic trace estimation, in Symposium on Simplicity in Algorithms (SOSA), SIAM, Philadelphia, 2021, pp. 142\u2013155, https:\/\/doi.org\/10.1137\/1.9781611976496.16.","DOI":"10.1137\/1.9781611976496.16"},{"key":"ref36","first-page":"543","volume-title":"Dokl. Akad. Nauk SSSR","volume":"269","author":"Nesterov Y. E.","year":"1983"},{"key":"ref37","volume-title":"Numerical Optimization","author":"Nocedal J.","year":"2006","edition":"2"},{"key":"ref38","unstructured":"R. Pascanu  and \nY. Bengio , Revisiting Natural Gradient for Deep Networks, preprint, arXiv:1301.3584, 2013."},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2007.11.026"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-246X.2006.02978.x"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(98)00116-6"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcp.2018.10.045"},{"key":"ref43","doi-asserted-by":"crossref","unstructured":"Y. Saad , Iterative Methods for Sparse Linear Systems, SIAM, Philadelphia, 2003, https:\/\/doi.org\/10.1137\/1.9780898718003.","DOI":"10.1137\/1.9780898718003"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.1162\/08997660260028683"},{"key":"ref45","first-page":"1646","volume":"33","author":"Shen Z.","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1090\/gsm\/058"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.1190\/1.3238367"},{"key":"ref48","doi-asserted-by":"crossref","unstructured":"P. Xu , \nF. Roosta , and \nM. W. Mahoney , Second-order optimization for non-convex machine learning: An empirical study, in Proceedings of the 2020 SIAM International Conference on Data Mining, SIAM, Philadelphia, 2020, pp. 199\u2013207, https:\/\/doi.org\/10.1137\/1.9781611976236.23.","DOI":"10.1137\/1.9781611976236.23"},{"key":"ref49","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017007"},{"key":"ref50","doi-asserted-by":"publisher","DOI":"10.1016\/j.cam.2021.113844"},{"key":"ref51","doi-asserted-by":"crossref","unstructured":"Z. Yao , \nA. Gholami , \nS. Shen , \nM. Mustafa , \nK. Keutzer , and \nM. Mahoney , ADAHESSIAN: An adaptive second order optimizer for machine learning, in Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 10665\u201310673.","DOI":"10.1609\/aaai.v35i12.17275"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.1007\/s10915-020-01367-x"},{"key":"ref53","unstructured":"A. Yu , \nY. Yang , and \nA. Townsend , A Quadrature Perspective on Frequency Bias in Neural Network Training with Nonuniform Data, preprint, arXiv:2205.14300, 2022."},{"key":"ref54","unstructured":"G. Zhang , \nJ. Martens , and \nR. B. Grosse , Fast convergence of natural gradient descent for over-parameterized neural networks, in Advances in Neural Information Processing Systems 32, Curran Associates, Red Hook, NY, 2019."},{"key":"ref55","unstructured":"B. Zhu , \nJ. Hu , \nY. Lou , and \nY. Yang , Implicit Regularization Effects of the Sobolev Norms in Image Processing, preprint, arXiv:2109.06255, 2021."}],"container-title":["SIAM Journal on Scientific Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/22M1477805","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T15:56:16Z","timestamp":1787327776000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/22M1477805"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,10]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,8,31]]}},"alternative-id":["10.1137\/22M1477805"],"URL":"https:\/\/doi.org\/10.1137\/22m1477805","relation":{},"ISSN":["1064-8275","1095-7197"],"issn-type":[{"value":"1064-8275","type":"print"},{"value":"1095-7197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,10]]}}}