{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,3]],"date-time":"2026-05-03T02:31:33Z","timestamp":1777775493476,"version":"3.51.4"},"reference-count":157,"publisher":"Emerald","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,6,15]]},"abstract":"<jats:p>The objective in a traditional reinforcement learning (RL) problem is to find a policy that optimizes the expected value of a performance metric such as the infinite-horizon cumulative discounted or long-run average cost\/reward. In practice, optimizing the expected value alone may not be satisfactory, in that it may be desirable to incorporate the notion of risk into the optimization problem formulation, either in the objective or as a constraint. Various risk measures have been proposed in the literature, e.g., exponential utility, variance, percentile performance, chance constraints, value at risk (quantile), conditional value-at-risk, prospect theory and its later enhancement, cumulative prospect theory.<\/jats:p>\n                  <jats:p>In this monograph, we consider risk-sensitive RL in two settings: one where the goal is to find a policy that optimizes the usual expected value objective while ensuring that a risk constraint is satisfied, and the other where the risk measure is the objective. We survey some of the recent work in this area specifically where policy gradient search is the solution approach. In the first risk-sensitive RL setting, we cover popular risk measures based on variance, conditional valueat-risk, and chance constraints, and present a template for policy gradient-based risk-sensitive RL algorithms using a Lagrangian formulation. For the setting where risk is incorporated directly into the objective function, we consider an exponential utility formulation, cumulative prospect theory, and coherent risk measures. This non-exhaustive survey aims to give a flavor of the challenges involved in solving risk-sensitive RL problems using policy gradient methods, as well as outlining some potential future research directions.<\/jats:p>","DOI":"10.1561\/2200000091","type":"journal-article","created":{"date-parts":[[2022,6,15]],"date-time":"2022-06-15T08:49:01Z","timestamp":1655282941000},"page":"537-693","source":"Crossref","is-referenced-by-count":21,"title":["Risk-Sensitive Reinforcement Learning via Policy Gradient Search"],"prefix":"10.1108","volume":"15","author":[{"given":"L. A.","family":"Prashanth","sequence":"first","affiliation":[{"name":"Indian Institute of Technology Madras ,","place":["India"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael C.","family":"Fu","sequence":"additional","affiliation":[{"name":"University of Maryland , ,","place":["College Park, USA"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","published-online":{"date-parts":[[2022,6,15]]},"reference":[{"issue":"11","key":"2026033012253618500_ref001","doi-asserted-by":"crossref","first-page":"1497","DOI":"10.1287\/mnsc.46.11.1497.12080","article-title":"Parameter-free elicitation of utility and probability weighting functions","volume":"46","author":"Abdellaoui","year":"2000","journal-title":"Management Science"},{"issue":"3","key":"2026033012253618500_ref002","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1137\/S0363012999361974","article-title":"Learning algorithms for Markov decision processes with average cost","volume":"40","author":"Abounadi","year":"2001","journal-title":"SIAM Journal on Control and Optimization"},{"issue":"1","key":"2026033012253618500_ref003","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1137\/S0363012998346621","article-title":"Stochastic approximation for nonexpansive maps: Application to Q-learning algorithms","volume":"41","author":"Abounadi","year":"2002","journal-title":"SIAM Journal on Control and Optimization"},{"issue":"7","key":"2026033012253618500_ref004","doi-asserted-by":"crossref","first-page":"1505","DOI":"10.1016\/S0378-4266(02)00281-9","article-title":"Spectral measures of risk: A coherent representation of subjective risk aversion","volume":"26","author":"Acerbi","year":"2002","journal-title":"Journal of Banking & Finance"},{"key":"2026033012253618500_ref005","first-page":"11","article-title":"Stochastic optimization","volume":"5","author":"Aleksandrov","year":"1968","journal-title":"Engineering Cybernetics"},{"key":"2026033012253618500_ref006","volume-title":"Constrained Markov Decision Processes","author":"Altman","year":"1999"},{"key":"2026033012253618500_ref007","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1137\/0331018","article-title":"Discrete-time controlled Markov processes with average cost criterion: A survey","volume":"31","author":"Arapostathis","year":"1993","journal-title":"SIAM Journal on Control and Optimization"},{"key":"2026033012253618500_ref008","volume-title":"Essays in the Theory of Risk Bearing","author":"Arrow","year":"1971"},{"issue":"3","key":"2026033012253618500_ref009","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1111\/1467-9965.00068","article-title":"Coherent measures of risk","volume":"9","author":"Artzner","year":"1999","journal-title":"Mathematical Finance"},{"key":"2026033012253618500_ref010","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-69033-9","volume-title":"Stochastic Simulation: Algorithms and Analysis","author":"Asmussen","year":"2007"},{"issue":"3","key":"2026033012253618500_ref011","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1007\/s11009-008-9089-z","article-title":"Properties of distortion risk measures","volume":"11","author":"Balb\u00e1s","year":"2009","journal-title":"Methodology and Computing in Applied Probability"},{"issue":"2","key":"2026033012253618500_ref012","doi-asserted-by":"crossref","first-page":"3892","DOI":"10.1214\/21-EJS1880","article-title":"Stochastic optimization with momentum: Convergence, fluctuations, and traps avoidance","volume":"15","author":"Barakat","year":"2021","journal-title":"Electronic Journal of Statistics"},{"key":"2026033012253618500_ref013","first-page":"173","article-title":"Thirty years of prospect theory in economics: A review and assessment","volume-title":"Journal of Economic Perspectives:","author":"Barberis","year":"2013"},{"issue":"3","key":"2026033012253618500_ref014","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1515\/MCMA.2009.011","article-title":"Computing VaR and CVaR using stochastic approximation and adaptive unconstrained importance sampling","volume":"15","author":"Bardou","year":"2009","journal-title":"Monte Carlo Methods and Applications"},{"key":"2026033012253618500_ref015","volume-title":"Infinite-horizon policy-gradient estimation","author":"Bartlett","year":"2011"},{"key":"2026033012253618500_ref016","first-page":"835","article-title":"Neuron-like elements that can solve difficult learning control problems","volume":"13","author":"Barto","year":"1983","journal-title":"IEEE Transaction on Systems, Man and Cybernetics"},{"issue":"4","key":"2026033012253618500_ref017","doi-asserted-by":"crossref","first-page":"880","DOI":"10.1287\/moor.1080.0324","article-title":"A learning algorithm for risk-sensitive cost","volume":"33","author":"Basu","year":"2008","journal-title":"Mathematics of Operations Research"},{"issue":"1","key":"2026033012253618500_ref018","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1287\/moor.2013.0601","article-title":"More risk-sensitive Markov decision processes","volume":"39","author":"B\u00e4uerle","year":"2014","journal-title":"Mathematics of Operations Research"},{"key":"2026033012253618500_ref019","volume-title":"Dynamic Programming","author":"Bellman","year":"1957"},{"key":"2026033012253618500_ref020","volume-title":"Dynamic Programming and Optimal Control, Vols. 1 & 2","author":"Bertsekas","year":"2007"},{"key":"2026033012253618500_ref021","volume-title":"Dynamic Programming and Optimal Control, Vol. II, 4th edition","author":"Bertsekas","year":"2012"},{"key":"2026033012253618500_ref022","volume-title":"Neuro-Dynamic Programming","author":"Bertsekas","year":"1996"},{"key":"2026033012253618500_ref023","first-page":"1691","volume-title":"A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation","author":"Bhandari","year":"2018"},{"key":"2026033012253618500_ref024","first-page":"11739","article-title":"Concentration of risk measures: A Wasserstein distance approach","volume-title":"Advances in Neural Information Processing Systems","author":"Bhat","year":"2019"},{"issue":"12","key":"2026033012253618500_ref025","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1016\/j.sysconle.2010.08.013","article-title":"An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes","volume":"59","author":"Bhatnagar","year":"2010","journal-title":"Systems & Control Letters"},{"issue":"70","key":"2026033012253618500_ref026","first-page":"1937","article-title":"A Simulation-Based Algorithm for Ergodic Control of Markov Chains Conditioned on Rare Events","volume":"7","author":"Bhatnagar","year":"2006","journal-title":"Journal of Machine Learning Research"},{"key":"2026033012253618500_ref027","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4471-4285-0","volume-title":"Stochastic Recursive Algorithms for Optimization","author":"Bhatnagar","year":"2013"},{"issue":"11","key":"2026033012253618500_ref028","doi-asserted-by":"crossref","first-page":"2471","DOI":"10.1016\/j.automatica.2009.07.008","article-title":"Natural actor-critic algorithms","volume":"45","author":"Bhatnagar","year":"2009","journal-title":"Automatica"},{"issue":"5","key":"2026033012253618500_ref029","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/S0167-6911(01)00152-9","article-title":"A sensitivity formula for risk-sensitive cost and the actor-critic algorithm","volume":"44","author":"Borkar","year":"2001","journal-title":"Systems & Control Letters"},{"issue":"2","key":"2026033012253618500_ref030","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1287\/moor.27.2.294.324","article-title":"Q-learning for risk-sensitive control","volume":"27","author":"Borkar","year":"2002","journal-title":"Mathematics of operations research"},{"issue":"3","key":"2026033012253618500_ref031","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.sysconle.2004.08.007","article-title":"An actor-critic algorithm for constrained Markov decision processes","volume":"54","author":"Borkar","year":"2005","journal-title":"Systems & Control Letters"},{"key":"2026033012253618500_ref032","doi-asserted-by":"crossref","DOI":"10.1007\/978-93-86279-38-5","volume-title":"Stochastic Approximation: A Dynamical Systems Viewpoint","author":"Borkar","year":"2008"},{"key":"2026033012253618500_ref033","volume-title":"Learning algorithms for risk-sensitive control","author":"Borkar","year":"2010"},{"key":"2026033012253618500_ref034","first-page":"2664","volume-title":"Risk-constrained Markov decision processes","author":"Borkar","year":"2010"},{"issue":"2","key":"2026033012253618500_ref035","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1137\/S0363012997331639","article-title":"The ODE method for convergence of stochastic approximation and reinforcement learning","volume":"38","author":"Borkar","year":"2000","journal-title":"SIAM Journal on Control and Optimization"},{"issue":"1","key":"2026033012253618500_ref036","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1287\/moor.27.1.192.334","article-title":"Risk-sensitive optimal control for Markov decision processes with monotone cost","volume":"27","author":"Borkar","year":"2002","journal-title":"Mathematics of Operations Research"},{"issue":"5","key":"2026033012253618500_ref037","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/S0167-6911(97)90015-3","article-title":"Stochastic approximation with two time scales","volume":"29","author":"Borkar","year":"1997","journal-title":"Systems & Control Letters"},{"issue":"2","key":"2026033012253618500_ref038","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1137\/16M1080173","article-title":"Optimization methods for large-scale machine learning","volume":"60","author":"Bottou","year":"2018","journal-title":"Siam Review"},{"issue":"3","key":"2026033012253618500_ref039","first-page":"395","article-title":"Les algorithmes stochastiques contournent-ils les pieges?","volume":"32","author":"Brandiere","year":"1996","journal-title":"Annales de l\u2019IHP Probabilit\u00e9s et Statistiques"},{"issue":"6","key":"2026033012253618500_ref040","doi-asserted-by":"crossref","first-page":"722","DOI":"10.1016\/j.orl.2007.01.001","article-title":"Large deviations bounds for estimating conditional value-at-risk","volume":"35","author":"Brown","year":"2007","journal-title":"Operations Research Letters"},{"issue":"4","key":"2026033012253618500_ref041","doi-asserted-by":"publisher","first-page":"937","DOI":"10.1287\/moor.20.4.937","article-title":"Optimal Investment Policies for a Firm With a Random Risk Process: Exponential Utility and Minimizing the Probability of Ruin","volume":"20","author":"Browne","year":"1995","journal-title":"Mathematics of Operations Research"},{"issue":"1","key":"2026033012253618500_ref042","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/BF00055711","article-title":"An experimental test of several generalized utility theories","volume":"2","author":"Camerer","year":"1989","journal-title":"Journal of Risk and Uncertainty"},{"key":"2026033012253618500_ref043","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/978-94-011-2952-7_9","volume-title":"Utility Theories: Measurements and Applications","author":"Camerer","year":"1992"},{"issue":"2","key":"2026033012253618500_ref044","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1007\/BF01065371","article-title":"Violations of the betweenness axiom and nonlinearity in probability","volume":"8","author":"Camerer","year":"1994","journal-title":"Journal of Risk and Uncertainty"},{"issue":"6","key":"2026033012253618500_ref045","doi-asserted-by":"crossref","first-page":"3935","DOI":"10.1137\/13093902X","article-title":"Risk-averse control of undiscounted transient Markov models","volume":"52","author":"Cavus","year":"2014","journal-title":"SIAM Journal on Control and Optimization"},{"key":"2026033012253618500_ref046","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-84628-690-2","volume-title":"Simulation-based Algorithms for Markov Decision Processes","author":"Chang","year":"2007"},{"key":"2026033012253618500_ref047","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4471-5022-0","volume-title":"Simulation-based Algorithms for Markov Decision Processes","author":"Chang","year":"2013"},{"key":"2026033012253618500_ref048","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1287\/mnsc.4.3.235","article-title":"Cost horizons and certainty equivalents: An approach to stochastic programming of heating oil","volume":"4","author":"Charnes","year":"1958","journal-title":"Management Science"},{"key":"2026033012253618500_ref049","volume-title":"Risk-averse Newsvendor Models","author":"Choi","year":"2009"},{"issue":"1","key":"2026033012253618500_ref050","first-page":"6070","article-title":"Risk-constrained reinforcement learning with percentile risk criteria","volume":"18","author":"Chow","year":"2017","journal-title":"The Journal of Machine Learning Research"},{"key":"2026033012253618500_ref051","first-page":"392","article-title":"Three variants on the Allais example","volume-title":"The American Economic Review:","author":"Conlisk","year":"1989"},{"key":"2026033012253618500_ref052","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1016\/S0005-1098(98)00153-8","article-title":"Risk-Sensitive and Minimax Control of Discrete-Time, Finite-State Markov Decision Processes","volume":"35","author":"Coraluppi","year":"1999","journal-title":"Automatica"},{"key":"2026033012253618500_ref053","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-1-4612-1784-8_2","volume-title":"Stochastic Analysis, Control, Optimization and Applications: A Volume in Honor of W. H. Fleming","author":"Coraluppi","year":"1999"},{"key":"2026033012253618500_ref054","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1109\/9.847737","article-title":"Mixed risk-neutral\/minimax control of discrete-time, finite-state Markov decision processes","volume":"45","author":"Coraluppi","year":"2000","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2026033012253618500_ref055","first-page":"3701","volume-title":"A tale of two-timescale reinforcement learning with the tightest finite-time bound","author":"Dalal","year":"2020"},{"key":"2026033012253618500_ref056","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.12079","volume-title":"Finite sample analyses for TD(0) with function approximation","author":"Dalal","year":"2018"},{"issue":"2","key":"2026033012253618500_ref057","doi-asserted-by":"crossref","first-page":"548","DOI":"10.1137\/S1052623402420528","article-title":"Optimization with stochastic dominance constraints","volume":"14","author":"Dentcheva","year":"2003","journal-title":"SIAM Journal on Optimization"},{"key":"2026033012253618500_ref058","volume-title":"Finite State Markovian Decision Processes","author":"Derman","year":"1970"},{"key":"2026033012253618500_ref059","doi-asserted-by":"crossref","first-page":"1418","DOI":"10.1109\/9.633830","article-title":"Risk-sensitive optimal control of hidden Markov models: Structural results","volume":"42","author":"Fern\u00e1ndez-Gaucherand","year":"1997","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"1","key":"2026033012253618500_ref060","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1287\/moor.14.1.147","article-title":"Variance-penalized Markov decision processes","volume":"14","author":"Filar","year":"1989","journal-title":"Mathematics of Operations Research"},{"issue":"1","key":"2026033012253618500_ref061","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1109\/9.362904","article-title":"Percentile performance criteria for limiting average Markov decision processes","volume":"40","author":"Filar","year":"1995","journal-title":"IEEE Transaction of Automatic Control"},{"issue":"6","key":"2026033012253618500_ref062","doi-asserted-by":"crossref","first-page":"1881","DOI":"10.1137\/S0363012993258720","article-title":"Risk-sensitive control on an infinite time horizon","volume":"33","author":"Fleming","year":"1995","journal-title":"SIAM Journal on Control and Optimization"},{"issue":"4","key":"2026033012253618500_ref063","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1007\/s007800200072","article-title":"Convex measures of risk and trading constraints","volume":"6","author":"F\u00f6llmer","year":"2002","journal-title":"Finance and stochastics"},{"key":"2026033012253618500_ref064","doi-asserted-by":"crossref","DOI":"10.1515\/9783110212075","volume-title":"Stochastic Finance: An Introduction in Discrete Time","author":"F\u00f6llmer","year":"2004"},{"key":"2026033012253618500_ref065","doi-asserted-by":"crossref","DOI":"10.1515\/9783110463453","volume-title":"Stochastic finance","author":"F\u00f6llmer","year":"2016"},{"key":"2026033012253618500_ref066","first-page":"575","volume-title":"Handbooks in Operations Research and Management Science: Simulation","author":"Fu","year":"2006"},{"key":"2026033012253618500_ref067","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4939-1384-8_5","article-title":"Stochastic Gradient Estimation","volume-title":"Handbook on Simulation Optimization","author":"Fu","year":"2015"},{"key":"2026033012253618500_ref068","first-page":"797","volume-title":"Escaping from saddle points-online stochastic gradient for tensor decomposition","author":"Ge","year":"2015"},{"issue":"4","key":"2026033012253618500_ref069","doi-asserted-by":"crossref","first-page":"2341","DOI":"10.1137\/120880811","article-title":"Stochastic first-and zeroth-order methods for nonconvex stochastic programming","volume":"23","author":"Ghadimi","year":"2013","journal-title":"SIAM Journal on Optimization"},{"key":"2026033012253618500_ref070","doi-asserted-by":"crossref","first-page":"366","DOI":"10.21236\/ADA197085","volume-title":"Likelilood ratio gradient estimation: an overview","author":"Glynn","year":"1987"},{"issue":"1","key":"2026033012253618500_ref071","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1006\/cogp.1998.0710","article-title":"On the shape of the probability weighting function","volume":"38","author":"Gonzalez","year":"1999","journal-title":"Cognitive psychology"},{"key":"2026033012253618500_ref072","first-page":"1941","volume-title":"Weighted bandits or: How bandits learn distorted values that are not expected","author":"Gopalan","year":"2017"},{"key":"2026033012253618500_ref073","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4757-3766-0","volume-title":"Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning","author":"Gosavi","year":"2003"},{"key":"2026033012253618500_ref074","first-page":"1968","article-title":"Variancereduced methods for machine learning","author":"Gower","year":"2020"},{"issue":"3","key":"2026033012253618500_ref075","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1016\/0167-2681(92)90017-6","article-title":"Predictions about indifference curves inside the unit triangle: A test of variants of expected utility theory","volume":"18","author":"Harless","year":"1992","journal-title":"Journal of Economic Behavior & Organization"},{"key":"2026033012253618500_ref076","article-title":"Measure-valued differentiation for stochastic processes: the finite horizon case","volume-title":"Tech. rep","author":"Heidergott","year":"2000"},{"issue":"3","key":"2026033012253618500_ref077","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/S0167-6911(96)00051-5","article-title":"Risk sensitive control of Markov processes in countable state space","volume":"29","author":"Hern\u00e1ndez-Hern\u00e1ndez","year":"1996","journal-title":"Systems & Control Letters"},{"key":"2026033012253618500_ref078","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/s002459900126","article-title":"Existence of risk sensitive optimal stationary policies for controlled Markov processes","volume":"40","author":"Hern\u00e1ndez-Hern\u00e1ndez","year":"1999","journal-title":"Applied Mathematics and Optimization"},{"key":"2026033012253618500_ref079","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1287\/mnsc.18.7.356","article-title":"Risk-sensitive Markov decision processes","volume":"18","author":"Howard","year":"1972","journal-title":"Management Science"},{"issue":"2","key":"2026033012253618500_ref080","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1287\/moor.1040.0129","article-title":"Robust dynamic programming","volume":"30","author":"Iyengar","year":"2005","journal-title":"Mathematics of Operations Research"},{"issue":"2","key":"2026033012253618500_ref081","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1287\/moor.2017.0872","article-title":"Risk-averse approximate dynamic programming with quantile-based risk measures","volume":"43","author":"Jiang","year":"2017","journal-title":"Mathematics of Operations Research"},{"issue":"9","key":"2026033012253618500_ref082","doi-asserted-by":"crossref","first-page":"2867","DOI":"10.1109\/TAC.2018.2822658","article-title":"Stochastic optimization in a cumulative prospect theory framework","volume":"63","author":"Jie","year":"2018","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2026033012253618500_ref083","first-page":"1724","volume-title":"How to escape saddle points efficiently","author":"Jin","year":"2017"},{"key":"2026033012253618500_ref084","first-page":"263","article-title":"Prospect theory: An analysis of decision under risk","volume-title":"Econometrica:","author":"Kahneman","year":"1979"},{"key":"2026033012253618500_ref085","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1214\/aoms\/1177729392","article-title":"Stochastic estimation of the maximum of a regression function","volume":"23","author":"Kiefer","year":"1952","journal-title":"Annals of Mathematical Statistics"},{"issue":"1","key":"2026033012253618500_ref086","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.orl.2018.11.005","article-title":"Concentration bounds for empirical conditional value-at-risk: The unbounded case","volume":"47","author":"Kolla","year":"2019","journal-title":"Operations Research Letters"},{"issue":"1","key":"2026033012253618500_ref087","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1137\/S036301299731669X","article-title":"Actor-Critic-Type Learning Algorithms for Markov Decision Processes","volume":"38","author":"Konda","year":"1999","journal-title":"SIAM Journal on control and Optimization"},{"issue":"2","key":"2026033012253618500_ref088","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1214\/105051604000000116","article-title":"Convergence rate of linear two-time-scale stochastic approximation","volume":"14","author":"Konda","year":"2004","journal-title":"The Annals of Applied Probability"},{"key":"2026033012253618500_ref089","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/978-1-4684-9352-8","volume-title":"Stochastic Approximation Methods for Constrained and Unconstrained Systems","author":"Kushner","year":"1978"},{"key":"2026033012253618500_ref090","volume-title":"PhD thesis","author":"Lin","year":"2013"},{"key":"2026033012253618500_ref091","doi-asserted-by":"crossref","DOI":"10.1016\/j.automatica.2018.07.028","article-title":"Probabilistically distorted risk-sensitive infinite-horizon dynamic programming","volume-title":"Automatica","author":"Lin","year":"2018"},{"key":"2026033012253618500_ref092","volume-title":"Cumulative weighting optimization: The discrete case","author":"Lin","year":"2013"},{"key":"2026033012253618500_ref093","volume-title":"Dynamic programming with non-convex risk-sensitive measures","author":"Lin","year":"2013"},{"issue":"3","key":"2026033012253618500_ref094","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1016\/j.ejor.2013.06.019","article-title":"Algorithmic aspects of mean-variance optimization in Markov decision processes","volume":"231","author":"Mannor","year":"2013","journal-title":"European Journal of Operational Research"},{"issue":"2","key":"2026033012253618500_ref095","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1109\/9.905687","article-title":"Simulation-based optimization of Markov reward processes","volume":"46","author":"Marbach","year":"2001","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2026033012253618500_ref096","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1007\/978-1-4612-4120-1_14","volume-title":"Systems and Control in the Twenty-First Century","author":"Marcus","year":"1997"},{"issue":"1","key":"2026033012253618500_ref097","first-page":"77","article-title":"Portfolio selection","volume":"7","author":"Markowitz","year":"1952","journal-title":"The Journal of Finance"},{"key":"2026033012253618500_ref098","volume-title":"Microeconomic theory","author":"Mas-Colell","year":"1995"},{"issue":"2","key":"2026033012253618500_ref099","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1023\/A:1017940631555","article-title":"Risk-sensitive reinforcement learning","volume":"49","author":"Mihatsch","year":"2002","journal-title":"Machine Learning"},{"key":"2026033012253618500_ref100","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1287\/opre.13.6.930","article-title":"Chance-constrained programming with joint constraints","volume":"13","author":"Miller","year":"1965","journal-title":"Operations Research"},{"key":"2026033012253618500_ref101","volume-title":"A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP","author":"Moharrami","year":"2022"},{"issue":"3","key":"2026033012253618500_ref102","doi-asserted-by":"crossref","first-page":"1671","DOI":"10.1214\/105051606000000448","article-title":"Convergence rate and averaging of nonlinear two-time-scale stochastic approximation algorithms","volume":"16","author":"Mokkadem","year":"2006","journal-title":"The Annals of Applied Probability"},{"issue":"4","key":"2026033012253618500_ref103","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1137\/050622328","article-title":"Convex Approximations of Chance Constrained Programs","volume":"17","author":"Nemirovski","year":"2007","journal-title":"SIAM Journal on Optimization"},{"key":"2026033012253618500_ref104","first-page":"4026","volume-title":"Stochastic variance-reduced policy gradient","author":"Papini","year":"2018"},{"issue":"2","key":"2026033012253618500_ref105","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1214\/aop\/1176990853","article-title":"Nonconvergence to unstable points in urn models and stochastic approximations","volume":"18","author":"Pemantle","year":"1990","journal-title":"The Annals of Probability"},{"key":"2026033012253618500_ref106","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1007\/BF02243227","article-title":"Sampling derivatives of probabilities","volume":"42","author":"Pflug","year":"1989","journal-title":"Computing"},{"key":"2026033012253618500_ref107","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4613-1449-3","volume-title":"Optimization of Stochastic Models","author":"Pflug","year":"1996"},{"issue":"4","key":"2026033012253618500_ref108","doi-asserted-by":"crossref","first-page":"838","DOI":"10.1137\/0330046","article-title":"Acceleration of stochastic approximation by averaging","volume":"30","author":"Polyak","year":"1992","journal-title":"SIAM Journal on Control and Optimization"},{"key":"2026033012253618500_ref109","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/978-3-319-11662-4_12","article-title":"Policy gradients for CVaR-constrained MDPs","volume-title":"Algorithmic Learning Theory (ALT)","author":"Prashanth","year":"2014"},{"issue":"5","key":"2026033012253618500_ref110","first-page":"2223","article-title":"Adaptive system optimization using random directions stochastic approximation","volume":"62","author":"Prashanth","year":"2018","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2026033012253618500_ref111","first-page":"252","article-title":"Actor-critic algorithms for risk-sensitive MDPs","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Prashanth","year":"2013"},{"issue":"3","key":"2026033012253618500_ref112","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1007\/s10994-016-5569-5","article-title":"Variance-constrained actor-critic algorithms for discounted and average reward MDPs","volume":"105","author":"Prashanth","year":"2016","journal-title":"Machine Learning"},{"key":"2026033012253618500_ref113","first-page":"5577","volume-title":"Concentration bounds for CVaR estimation: The cases of light-tailed and heavy-tailed distributions","author":"Prashanth","year":"2020"},{"key":"2026033012253618500_ref114","first-page":"1406","volume-title":"Cumulative prospect theory meets reinforcement learning: prediction and control","author":"Prashanth","year":"2016"},{"issue":"3","key":"2026033012253618500_ref115","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1007\/s10994-020-05912-5","article-title":"Concentration bounds for temporal difference learning with linear function approximation: The case of batch data and uniform sampling","volume":"110","author":"Prashanth","year":"2021","journal-title":"Mach. Learn"},{"key":"2026033012253618500_ref116","volume-title":"Stochastic Programming","author":"Prekopa","year":"2003"},{"key":"2026033012253618500_ref117","first-page":"113","volume-title":"On probabilistic constrained programming","author":"Pr\u00e9kopa","year":"1970"},{"key":"2026033012253618500_ref118","first-page":"497","article-title":"The probability weighting function","volume-title":"Econometrica:","author":"Prelec","year":"1998"},{"key":"2026033012253618500_ref119","doi-asserted-by":"crossref","DOI":"10.1002\/9780470316887","volume-title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","author":"Puterman","year":"1994"},{"key":"2026033012253618500_ref120","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1287\/opre.37.5.830","article-title":"Sensitivity analysis for simulations via likelihood ratios","volume":"37","author":"Reiman","year":"1989","journal-title":"Operations Research"},{"key":"2026033012253618500_ref121","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.spa.2004.03.004","article-title":"Dynamic coherent risk measures","volume":"112","author":"Riedel","year":"2004","journal-title":"Stochastic Processes and Their Applications"},{"key":"2026033012253618500_ref122","first-page":"400","article-title":"A stochastic approximation method","volume-title":"The Annals of Mathematical Statistics:","author":"Robbins","year":"1951"},{"key":"2026033012253618500_ref123","doi-asserted-by":"crossref","first-page":"21","DOI":"10.21314\/JOR.2000.038","article-title":"Optimization of conditional value-at-risk","volume":"2","author":"Rockafellar","year":"2000","journal-title":"Journal of Risk"},{"key":"2026033012253618500_ref124","volume-title":"Introduction to Stochastic Dynamic Programming","author":"Ross","year":"1983"},{"key":"2026033012253618500_ref125","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1287\/opre.37.1.72","article-title":"Sensitivity analysis of computer simulation models via the score efficient","volume":"37","author":"Rubinstein","year":"1989","journal-title":"Operations Research"},{"key":"2026033012253618500_ref126","first-page":"503","article-title":"Stochastic approximation","volume-title":"Handbook of Sequential Analysis:","author":"Ruppert","year":"1991"},{"key":"2026033012253618500_ref127","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1007\/s10107-010-0393-3","article-title":"Risk-averse dynamic programming for Markov decision processes","volume":"125","author":"Ruszczy\u0144ski","year":"2010","journal-title":"Mathematical Programming"},{"issue":"3","key":"2026033012253618500_ref128","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1287\/moor.1060.0204","article-title":"Conditional risk mappings","volume":"31","author":"Ruszczy\u0144ski","year":"2006","journal-title":"Mathematics of Operations Research"},{"key":"2026033012253618500_ref129","volume-title":"Approximation theorems of mathematical statistics","author":"Serfling","year":"2009"},{"key":"2026033012253618500_ref130","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611973433","volume-title":"Lectures on Stochastic Programming: Modeling and Theory","author":"Shapiro","year":"2014"},{"issue":"5","key":"2026033012253618500_ref131","doi-asserted-by":"crossref","first-page":"3652","DOI":"10.1137\/120899005","article-title":"Risk-sensitive Markov control processes","volume":"51","author":"Shen","year":"2013","journal-title":"SIAM Journal on Control and Optimization"},{"key":"2026033012253618500_ref132","first-page":"5729","volume-title":"Hessian aided policy gradient","author":"Shen","year":"2019"},{"issue":"1","key":"2026033012253618500_ref133","doi-asserted-by":"crossref","first-page":"171","DOI":"10.2140\/pjm.1958.8.171","article-title":"On general minimax theorems","volume":"8","author":"Sion","year":"1958","journal-title":"Pacific J. Math"},{"key":"2026033012253618500_ref134","first-page":"794","article-title":"The variance of discounted Markov decision processes","volume-title":"Journal of Applied Probability:","author":"Sobel","year":"1982"},{"issue":"1","key":"2026033012253618500_ref135","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1007\/BF01075236","article-title":"A test of generalized expected utility theory","volume":"35","author":"Sopher","year":"1993","journal-title":"Theory and Decision"},{"key":"2026033012253618500_ref136","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1109\/9.119632","article-title":"Multivariate stochastic approximation using simultaneous perturbation gradient approximation","volume":"37","author":"Spall","year":"1992","journal-title":"IEEE Transactions on Automatic Control"},{"key":"2026033012253618500_ref137","first-page":"2803","volume-title":"Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning","author":"Srikant","year":"2019"},{"key":"2026033012253618500_ref138","first-page":"332","article-title":"Developments in non-expected utility theory: The hunt for a descriptive theory of choice under risk","volume-title":"Journal of Economic Literature:","author":"Starmer","year":"2000"},{"key":"2026033012253618500_ref139","volume-title":"PhD thesis","author":"Sutton","year":"1984"},{"issue":"1","key":"2026033012253618500_ref140","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1023\/A:1022633531479","article-title":"Learning to predict by the methods of temporal differences","volume":"3","author":"Sutton","year":"1988","journal-title":"Machine Learning"},{"key":"2026033012253618500_ref141","first-page":"1057","article-title":"Policy gradient methods for reinforcement learning with function approximation","volume":"99","author":"Sutton","year":"1999","journal-title":"NIPS"},{"key":"2026033012253618500_ref142","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018","edition":"2nd"},{"key":"2026033012253618500_ref143","doi-asserted-by":"crossref","DOI":"10.1002\/9780470400531.eorms0714","article-title":"Reinforcement learning algorithms for MDPs","volume-title":"Wiley Encyclopedia of Operations Research and Management Science","author":"Szepesv\u00e1ri","year":"2011"},{"key":"2026033012253618500_ref144","volume-title":"Robust, Risk-sensitive, and Data-driven Control of Markov Decision Processes","author":"Tallec","year":"2007"},{"key":"2026033012253618500_ref145","first-page":"387","volume-title":"Policy gradients with variance related risk criteria","author":"Tamar","year":"2012"},{"key":"2026033012253618500_ref146","first-page":"1468","article-title":"Policy gradient for coherent risk measures","volume":"28","author":"Tamar","year":"2015","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026033012253618500_ref147","article-title":"Policy gradient for coherent risk measures","volume-title":"CoRR","author":"Tamar","year":"2015"},{"key":"2026033012253618500_ref148","first-page":"495","volume-title":"Temporal difference methods for the variance of the reward to go","author":"Tamar","year":"2013"},{"key":"2026033012253618500_ref149","volume-title":"Optimizing the CVaR via sampling","author":"Tamar","year":"2014"},{"key":"2026033012253618500_ref150","volume-title":"Policy gradients beyond expectations: Conditional Value-at-Risk","author":"Tamar","year":"2014"},{"key":"2026033012253618500_ref151","first-page":"6225","volume-title":"Concentration Inequalities for Conditional Value at Risk","author":"Thomas","year":"2019"},{"issue":"5","key":"2026033012253618500_ref152","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/9.580874","article-title":"An analysis of temporal-difference learning with function approximation","volume":"42","author":"Tsitsiklis","year":"1997","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"11","key":"2026033012253618500_ref153","doi-asserted-by":"crossref","first-page":"1799","DOI":"10.1016\/S0005-1098(99)00099-0","article-title":"Average cost temporal-difference learning","volume":"35","author":"Tsitsiklis","year":"1999","journal-title":"Automatica"},{"issue":"4","key":"2026033012253618500_ref154","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/BF00122574","article-title":"Advances in prospect theory: Cumulative representation of uncertainty","volume":"5","author":"Tversky","year":"1992","journal-title":"Journal of Risk and Uncertainty"},{"issue":"3","key":"2026033012253618500_ref155","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1016\/j.orl.2009.11.008","article-title":"Deviation inequalities for an estimator of the conditional value-at-risk","volume":"38","author":"Wang","year":"2010","journal-title":"Operations Research Letters"},{"key":"2026033012253618500_ref156","volume-title":"Risk-sensitive Optimal Control. Wiley-Interscience series in systems and optimization","author":"Whittle","year":"1990"},{"issue":"6","key":"2026033012253618500_ref157","doi-asserted-by":"crossref","first-page":"3586","DOI":"10.1137\/19M1288012","article-title":"Global convergence of policy gradient methods to (almost) locally optimal policies","volume":"58","author":"Zhang","year":"2020","journal-title":"SIAM Journal on Control and Optimization"}],"container-title":["Foundations and Trends\u00ae in Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/ftmal\/article-pdf\/15\/5\/537\/11147241\/2200000091en.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/ftmal\/article-pdf\/15\/5\/537\/11147241\/2200000091en.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T18:10:54Z","timestamp":1777486254000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/ftmal\/article\/15\/5\/537\/1332158\/Risk-Sensitive-Reinforcement-Learning-via-Policy"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,15]]},"references-count":157,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,6,15]]}},"URL":"https:\/\/doi.org\/10.1561\/2200000091","relation":{},"ISSN":["1935-8237","1935-8245"],"issn-type":[{"value":"1935-8237","type":"print"},{"value":"1935-8245","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,15]]}}}