{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T23:18:57Z","timestamp":1792279137818,"version":"4.4.0"},"posted":{"date-parts":[[2026]]},"group-title":"SSRN","reference-count":24,"publisher":"Elsevier BV","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>Conditional value-at-risk (CVaR) is a prominent risk measure in financial engineering, energy systems, and supply chain management. In these domains, Markov decision processes (MDPs) with a long-run CVaR criterion effectively mitigate cost variability over a specified horizon. However, implementing MDPs relies on known transition models, which are typically unavailable in practice. This necessitates a model-free approach to risk-sensitive dynamic optimization. To tackle this challenge, we propose a reinforcement learning algorithm that simultaneously conducts policy evaluation and improvement based on a CVaR-specific Bellman local optimality equation. This algorithm employs a nonparametric incremental learning approach for policy improvement, relying on a single sample trajectory to identify the optimal policy. Under appropriate technical conditions, we prove almost sure convergence of the algorithm and derive its convergence rate. Our analysis reveals that the optimal convergence rate, measured by the mean absolute error of policy estimators, is of order O(1\/n). Our main algorithm and results are further extended to solving the mean-CVaR optimization problem. Numerical experiments corroborate these results.<\/jats:p>","DOI":"10.2139\/ssrn.6386418","type":"posted-content","created":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T08:01:46Z","timestamp":1774944106000},"source":"Crossref","is-referenced-by-count":1,"title":["Long-Run Conditional Value-at-Risk Reinforcement Learning"],"prefix":"10.2139","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-2302-8407","authenticated-orcid":true,"given":"Qixin","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Cao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2989-8048","authenticated-orcid":true,"given":"Jianqiang","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingjie","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li","family":"Xia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"78","reference":[{"issue":"3","key":"ref1","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1137\/S0363012999361974","article-title":"Learning algorithms for Markov decision processes with average cost","volume":"40","author":"J Abounadi","year":"2001","journal-title":"SIAM Journal on Control and Optimization"},{"key":"ref2","doi-asserted-by":"crossref","DOI":"10.1016\/j.artint.2022.103845","article-title":"Risk-averse optimization of reward-based coherent risk measures","volume":"316","author":"M Bonetti","year":"2023","journal-title":"Artificial Intelligence"},{"key":"ref3","doi-asserted-by":"crossref","DOI":"10.1007\/978-93-86279-38-5","volume":"9","author":"V S Borkar","year":"2008","journal-title":"Stochastic Approximation: A Dynamical Systems Viewpoint"},{"key":"ref4","article-title":"Learning algorithms for risk-sensitive control","volume":"5","author":"V S Borkar","year":"2010","journal-title":"Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems-MTNS"},{"issue":"6","key":"ref5","doi-asserted-by":"crossref","first-page":"722","DOI":"10.1016\/j.orl.2007.01.001","article-title":"Large deviations bounds for estimating conditional value-at-risk","volume":"35","author":"D B Brown","year":"2007","journal-title":"Operations Research Letters"},{"key":"ref6","article-title":"Provably efficient iterated CVaR reinforcement learning with function approximation and human feedback","author":"Y Chen","year":"2024","journal-title":"The Twelfth International Conference on Learning Representations"},{"issue":"167","key":"ref7","first-page":"1","article-title":"Risk-constrained reinforcement learning with percentile risk criteria","volume":"18","author":"Y Chow","year":"2018","journal-title":"Journal of Machine Learning Research"},{"key":"ref8","first-page":"1522","article-title":"Risk-sensitive and robust decision-making: A CVaR optimization approach","volume":"1","author":"Y Chow","year":"2015","journal-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems"},{"issue":"2","key":"ref9","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1287\/mnsc.1080.0901","article-title":"Simulating sensitivities of conditional value at risk","volume":"55","author":"L J Hong","year":"2009","journal-title":"Management Science"},{"issue":"6","key":"ref10","doi-asserted-by":"crossref","first-page":"2889","DOI":"10.1287\/ijoc.2022.1214","article-title":"A stochastic approximation method for simulation-based quantile optimization","volume":"34","author":"J Hu","year":"2022","journal-title":"INFORMS Journal on Computing"},{"issue":"3","key":"ref11","doi-asserted-by":"crossref","first-page":"1535","DOI":"10.1287\/opre.2022.0534","article-title":"Quantile optimization via multiple-timescale local search for black-box functions","volume":"73","author":"J Hu","year":"2025","journal-title":"Operations Research"},{"issue":"10","key":"ref12","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1080\/24725854.2024.2429714","article-title":"Simulation optimization of conditional value-at-risk","volume":"57","author":"J Hu","year":"2025","journal-title":"IISE Transactions"},{"issue":"2","key":"ref13","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1287\/moor.2017.0872","article-title":"Risk-averse approximate dynamic programming with quantile-based risk measures","volume":"43","author":"D R Jiang","year":"2018","journal-title":"Mathematics of Operations Research"},{"key":"ref14","author":"H J Kushner","year":"2003","journal-title":"Stochastic Approximation and Recursive Algorithms and Applications"},{"issue":"2","key":"ref15","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1287\/moor.2015.0747","article-title":"Time-consistent decisions and temporal decomposition of coherent risk functionals","volume":"41","author":"G C Pflug","year":"2016","journal-title":"Mathematics of Operations Research"},{"key":"ref16","author":"L A Prashanth","year":"2022","journal-title":"Risk-Sensitive Reinforcement Learning via Policy Gradient Search"},{"key":"ref17","doi-asserted-by":"crossref","first-page":"21","DOI":"10.21314\/JOR.2000.038","article-title":"Optimization of conditional value-at-risk","volume":"2","author":"R T Rockafellar","year":"2000","journal-title":"Journal of Risk"},{"key":"ref18","first-page":"431","article-title":"Burning RED: Unlocking subtask-driven reinforcement learning and risk-awareness in average-reward Markov decision processes","volume":"6","author":"J S Rojas","year":"2025","journal-title":"Reinforcement Learning Journal"},{"key":"ref19","first-page":"333","article-title":"CVaR Q-learning","author":"S Stanko","year":"2019","journal-title":"Computational Intelligence: 11th International Joint Conference"},{"key":"ref20","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v29i1.9561","article-title":"Optimizing the CVaR via sampling","volume":"29","author":"A Tamar","year":"2015","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"ref21","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1007\/978-1-4757-6594-6_17","author":"S Uryasev","year":"2001","journal-title":"Conditional value-at-risk: Optimization approach. Stochastic Optimization: Algorithms and Applications"},{"key":"ref22","author":"Q Wang","year":"2025","journal-title":"Statistical inference in conditional value-at-risk optimization. SSRN URL"},{"issue":"12","key":"ref23","doi-asserted-by":"crossref","first-page":"4049","DOI":"10.1111\/poms.14077","article-title":"Risk-sensitive Markov decision processes with long-run CVaR criterion","volume":"32","author":"L Xia","year":"2023","journal-title":"Production and Operations Management"},{"issue":"10","key":"ref24","doi-asserted-by":"crossref","first-page":"6546","DOI":"10.1109\/TAC.2024.3371380","article-title":"Relative Q-learning for average-reward Markov decision processes with continuous states","volume":"69","author":"X Yang","year":"2024","journal-title":"IEEE Transactions on Automatic Control"}],"container-title":[],"original-title":[],"deposited":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T14:50:04Z","timestamp":1791557404000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ssrn.com\/abstract=6386418"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":24,"URL":"https:\/\/doi.org\/10.2139\/ssrn.6386418","relation":{},"subject":[],"published":{"date-parts":[[2026]]},"subtype":"preprint"}}