{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T14:25:31Z","timestamp":1787322331522,"version":"build-2736575974"},"reference-count":52,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"4","funder":[{"name":"NSF CAREER","award":["DMS-2339240"],"award-info":[{"award-number":["DMS-2339240"]}]},{"name":"JP Morgan Faculty Research Award"},{"DOI":"10.13039\/100027065","name":"Google Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100027065","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Control Optim."],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:p>Abstract.<\/jats:p>\n                  <jats:p>Nonlinear control systems with partial information to the decision maker are prevalent in a variety of applications. As a step toward studying such nonlinear systems, this work explores reinforcement learning methods for finding the optimal policy in the nearly linear-quadratic regulator systems. In particular, we consider a dynamic system that combines linear and nonlinear components, and is governed by a policy with the same structure. Assuming that the nonlinear component comprises kernels with small Lipschitz coefficients, we characterize the optimization landscape of the cost function. Although the cost function is nonconvex in general, we establish the local strong convexity and smoothness in the vicinity of the global optimizer. Additionally, we propose an initialization mechanism to leverage these properties. Building on the developments, we design a policy gradient algorithm that is guaranteed to converge to the globally optimal policy with a linear rate.<\/jats:p>","DOI":"10.1137\/23m1560781","type":"journal-article","created":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T07:35:37Z","timestamp":1755502537000},"page":"2936-2963","source":"Crossref","is-referenced-by-count":1,"title":["Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-Quadratic Regulators"],"prefix":"10.1137","volume":"63","author":[{"given":"Yinbin","family":"Han","sequence":"first","affiliation":[{"name":"Department of Finance and Risk Engineering, Tandon School of Engineering, New York University, Brooklyn, NY 11201 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meisam","family":"Razaviyayn","sequence":"additional","affiliation":[{"name":"Daniel J. Epstein Department of Industrial and Systems Engineering, University of Southern California, Los Angeles, CA 90089-0193 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4293-3450","authenticated-orcid":true,"given":"Renyuan","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Finance and Risk Engineering, Tandon School of Engineering, New York University, Brooklyn, NY 11201 USA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2025,8,18]]},"reference":[{"key":"ref1","first-page":"1","volume":"22","author":"Agarwal A.","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref2","volume-title":"Optimal Control: Linear Quadratic Methods","author":"Anderson B. D.","year":"2007"},{"key":"ref3","volume-title":"Dynamic Programming and Optimal Control","volume":"1","author":"Bertsekas D.","year":"2012"},{"key":"ref4","unstructured":"J. Bhandari and D. Russo, On the linear convergence of policy gradient methods for finite MDPs, in International Conference on Artificial Intelligence and Statistics, PMLR, 2021, pp. 2386\u20132394."},{"key":"ref5","unstructured":"J. Bu, A. Mesbahi, M. Fazel, and M. Mesbahi, LQR through the Lens of First Order Methods: Discrete-time Case, preprint, https:\/\/arxiv.org\/abs\/1907.08921, 2019."},{"key":"ref6","first-page":"2563","author":"Cen S.","year":"2021","journal-title":"Oper. Res."},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1007\/s10208-019-09426-y"},{"key":"ref8","first-page":"653","volume":"28","author":"Deng Y.","year":"2016","journal-title":"Ieee T. Neur. Net. Lear."},{"key":"ref9","first-page":"8378","volume-title":"NeurIPS","author":"Ding D.","year":"2020"},{"key":"ref10","first-page":"4809","volume":"17","author":"Farahmand A.-m.","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref11","unstructured":"I. Fatkhullin, A. Barakat, A. Kireeva, and N. He, Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-Non-degenerate Policies, preprint, https:\/\/arxiv.org\/abs\/2302.01734, 2023."},{"key":"ref12","unstructured":"M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, Global convergence of policy gradient methods for the linear quadratic regulator, in International Conference on Machine Learning, PMLR, 2018, pp. 1467\u20131476."},{"key":"ref13","unstructured":"A. D. Flaxman, A. T. Kalai, and H. B. McMahan, Online convex optimization in the bandit setting: Gradient descent without a gradient, in Proceedings of the Sixteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA \u201905, 2005, Society for Industrial and Applied Mathematics, Philadelphia, pp. 385\u2013394."},{"key":"ref14","unstructured":"Z. Fu, Z. Yang, and Z. Wang, Single-timescale actor-critic provably finds globally optimal policy, in International Conference on Learning Representations, 2021, https:\/\/openreview.net\/forum?id=pqZV_srUVmK."},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2020.3037046"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2011.2104999"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1137\/20M1382386"},{"key":"ref18","unstructured":"Y. Han, M. Razaviyayn, and R. Xu, Policy Gradient Converges to the Globally Optimal Policy for Nearly Linear-quadratic Regulators, preprint, https:\/\/arxiv.org\/abs\/2303.08431, 2023."},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1017\/CCOL0521382483"},{"key":"ref20","doi-asserted-by":"crossref","unstructured":"J. P. Jansch-Porto, B. Hu, and G. E. Dullerud, Convergence guarantees of policy optimization methods for Markovian jump linear systems, in 2020 American Control Conference (ACC), IEEE, 2020, pp. 2882\u20132887.","DOI":"10.23919\/ACC45564.2020.9147571"},{"key":"ref21","unstructured":"Z. Jin, J. M. Schmitt, and Z. Wen, On the Analysis of Model-Free Methods for the Linear Quadratic Regulator, preprint, https:\/\/arxiv.org\/abs\/2007.03861, 2020."},{"key":"ref22","first-page":"583","volume-title":"Artificial Intelligence and Statistics, PMLR","author":"Kar P.","year":"2012"},{"key":"ref23","first-page":"1334","volume":"17","author":"Levine S.","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2019.2901089"},{"key":"ref25","unstructured":"T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning, in International Conference on Learning Representations, 2016, http:\/\/arxiv.org\/abs\/1509.02971."},{"key":"ref26","first-page":"10565","volume":"32","author":"Liu B.","year":"2019","journal-title":"Adv. Neur. In"},{"key":"ref27","first-page":"7624","volume-title":"NeurIPS","author":"Liu Y.","year":"2020"},{"key":"ref28","first-page":"51","volume":"21","author":"Malik D.","year":"2019","journal-title":"Journal of Machine Leanring Research."},{"key":"ref29","first-page":"10154","volume":"32","author":"Mania H.","year":"2019","journal-title":"Adv. Neur. In"},{"key":"ref30","unstructured":"V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, Playing Atari with Deep Reinforcement Learning, preprint, https:\/\/arxiv.org\/abs\/1312.5602, 2013."},{"key":"ref31","doi-asserted-by":"crossref","unstructured":"H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovi\u0107, Global exponential convergence of gradient methods over the nonconvex landscape of the linear quadratic regulator, in 2019 Proceedings of the IEEE 58th Conference on Decision and Control (CDC), IEEE, 2019, pp. 7474\u20137479.","DOI":"10.1109\/CDC40024.2019.9029985"},{"key":"ref32","doi-asserted-by":"crossref","unstructured":"G. Qu, C. Yu, S. Low, and A. Wierman, Exploiting linear models for model-free nonlinear control: A provably convergent policy gradient approach, in 2021 Proceedings of the IEEE 60th Conference on Decision and Control (CDC), IEEE, 2021, pp. 6539\u20136546.","DOI":"10.1109\/CDC45484.2021.9683735"},{"key":"ref33","volume-title":"Nonlinear Systems: Analysis, Stability, and Control","volume":"10","author":"Sastry S.","year":"2013"},{"key":"ref34","doi-asserted-by":"crossref","unstructured":"B. Sch\u00f6lkopf, A. Smola, and K.R. M\u00fcller, Kernel principal component analysis, in International Conference on Artificial Neural Networks, Springer, 1997, pp. 583\u2013588.","DOI":"10.1007\/BFb0020217"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-77242-4"},{"key":"ref37","first-page":"1057","volume":"12","author":"Sutton R. S.","year":"1999","journal-title":"Adv. Neur. In"},{"key":"ref38","unstructured":"L. Szpruch, T. Treetanthiploet, and Y. Zhang, Exploration-exploitation Trade-Off for Continuous-Time Episodic Reinforcement Learning with Linear-Convex Models, preprint, https:\/\/arxiv.org\/abs\/2112.10264, 2021."},{"key":"ref39","unstructured":"L. Szpruch, T. Treetanthiploet, and Y. Zhang, Optimal Scheduling of Entropy Regulariser for Continuous-Time Linear-Quadratic Reinforcement Learning, preprint, https:\/\/arxiv.org\/abs\/2208.04466, 2022."},{"key":"ref40","doi-asserted-by":"crossref","unstructured":"J. Umlauft, T. Beckers, M. Kimmel, and S. Hirche, Feedback linearization using Gaussian processes, in 2017 Proceedings of the IEEE 56th Annual Conference on Decision and Control (CDC), 2017, pp. 5249\u20135255.","DOI":"10.1109\/CDC.2017.8264435"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2019.2958840"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511794308.006"},{"key":"ref43","first-page":"8145","volume":"21","author":"Wang H.","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref44","unstructured":"L. Wang, Q. Cai, Z. Yang, and Z. Wang, Neural policy gradient methods: Global optimality and rates of convergence, in International Conference on Learning Representations, 2020, https:\/\/openreview.net\/forum?id=BJgQfkSYDS."},{"key":"ref45","doi-asserted-by":"crossref","unstructured":"T. Westenbroek, D. Fridovich-Keil, E. Mazumdar, S. Arora, V. Prabhu, S. S. Sastry, and C. J. Tomlin, Feedback linearization for uncertain systems via reinforcement learning, in 2020 Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2020, pp. 1364\u20131371.","DOI":"10.1109\/ICRA40945.2020.9197158"},{"key":"ref46","first-page":"1","volume":"23","author":"Xiao L.","year":"2022","journal-title":"J. Mach. Learn. Res."},{"key":"ref47","first-page":"11581","volume-title":"ICML","author":"Xu T.","year":"2021"},{"key":"ref48","first-page":"8353","volume":"32","author":"Yang Z.","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref49","doi-asserted-by":"publisher","DOI":"10.1016\/j.swevo.2018.04.008"},{"key":"ref50","first-page":"4572","volume":"33","author":"Zhang J.","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref51","doi-asserted-by":"publisher","DOI":"10.1137\/20M1347942"},{"key":"ref52","unstructured":"Y. Zhang, Z. Yang, and Z. Wang, Provably efficient actor-critic for risk-sensitive and robust adversarial RL: A linear-quadratic case, in International Conference on Artificial Intelligence and Statistics, PMLR, 2021, pp. 2764\u20132772."}],"container-title":["SIAM Journal on Control and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/23M1560781","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T13:20:45Z","timestamp":1787318445000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/23M1560781"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,18]]},"references-count":52,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,8,31]]}},"alternative-id":["10.1137\/23M1560781"],"URL":"https:\/\/doi.org\/10.1137\/23m1560781","relation":{},"ISSN":["0363-0129","1095-7138"],"issn-type":[{"value":"0363-0129","type":"print"},{"value":"1095-7138","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,18]]}}}