{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T14:59:34Z","timestamp":1787324374914,"version":"build-2736575974"},"reference-count":40,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM J. Control Optim."],"published-print":{"date-parts":[[2015,1]]},"abstract":"<jats:p>In classical Markov decision process (MDP) theory, we search for a policy that, say, minimizes the expected infinite horizon discounted cost. Expectation is, of course, a risk neutral measure, which does not suffice in many applications, particularly in finance. We replace the expectation with a general risk functional, and call such models risk-aware MDP models. We consider minimization of such risk functionals in two cases, the expected utility framework, and conditional value-at-risk, a popular coherent risk measure. Later, we consider risk-aware MDPs wherein the risk is expressed in the constraints. This includes stochastic dominance constraints, and the classical chance-constrained optimization problems. In each case, we develop a convex analytic approach to solve such risk-aware MDPs. In most cases, we show that the problem can be formulated as an infinite-dimensional linear program (LP) in occupation measures when we augment the state space. We provide a discretization method and finite approximations for solving the resulting LPs. A striking result is that the chance-constrained MDP problem can be posed as an LP via the convex analytic method.<\/jats:p>","DOI":"10.1137\/140969221","type":"journal-article","created":{"date-parts":[[2015,6,18]],"date-time":"2015-06-18T14:14:13Z","timestamp":1434636853000},"page":"1569-1598","source":"Crossref","is-referenced-by-count":43,"title":["A Convex Analytic Approach to Risk-Aware Markov Decision Processes"],"prefix":"10.1137","volume":"53","author":[{"given":"William B.","family":"Haskell","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rahul","family":"Jain","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"351","published-online":{"date-parts":[[2015,6,18]]},"reference":[{"key":"atypb1","unstructured":"E. Altman,\n                      Constrained Markov Decision Processes\n                      , Stoch. Model. 7, Chapman and Hall, CRC, Boca Raton, FL, 1999."},{"key":"atypb2","unstructured":"E. J. Anderson and P. Nash,\n                      Linear Programming in Infinite-Dimensional Spaces\n                      , John Wiley & Sons, Chichester, England, 1987."},{"key":"atypb3","doi-asserted-by":"publisher","DOI":"10.1007\/s00186-011-0367-0"},{"key":"atypb4","doi-asserted-by":"publisher","DOI":"10.1287\/moor.2013.0601"},{"key":"atypb5","doi-asserted-by":"publisher","DOI":"10.1007\/BF00353877"},{"key":"atypb6","first-page":"347","author":"Borkar V. S.","year":"2002","journal-title":"Boston"},{"key":"atypb7","doi-asserted-by":"crossref","unstructured":"V. Borkar and R. Jain,\n                      Risk-constrained markov decision processes\n                      , in Proceedings of the IEEE Control and Decision Conference, IEEE, Piscataway, NJ, 2010.","DOI":"10.1109\/CDC.2010.5718076"},{"key":"atypb8","first-page":"10","volume":"19","author":"Bronshtein E. M.","year":"1978","journal-title":"Sibirsk. Mat. Zh."},{"key":"atypb9","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2013.1251"},{"key":"atypb10","doi-asserted-by":"publisher","DOI":"10.1137\/S1052623402420528"},{"key":"atypb11","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-003-0453-z"},{"key":"atypb12","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-007-0165-x"},{"key":"atypb13","unstructured":"C. Derman,\n                      Finite State Markovian Decision Processes\n                      , Academic Press, Orlando, FL, 1970."},{"key":"atypb14","doi-asserted-by":"publisher","DOI":"10.1137\/S0363012997320614"},{"key":"atypb15","doi-asserted-by":"publisher","DOI":"10.1287\/moor.14.1.147"},{"key":"atypb16","doi-asserted-by":"crossref","unstructured":"H. Follmer and A. Schied,\n                      Stochastic Finance: An Introduction in Discrete Time\n                      , Walter de Gruyter, Berlin, 2004.","DOI":"10.1515\/9783110212075"},{"key":"atypb17","doi-asserted-by":"publisher","DOI":"10.1137\/120874679"},{"key":"atypb18","doi-asserted-by":"publisher","DOI":"10.1007\/s001860000071"},{"key":"atypb19","doi-asserted-by":"publisher","DOI":"10.1137\/S0363012999361627"},{"key":"atypb20","doi-asserted-by":"crossref","unstructured":"O. Hernandez-Lerma and J. B. Lasserre,\n                      Discrete-Time Markov Control Processes: Basic Optimality Criteria\n                      , Springer-Verlag, New York, 1996.","DOI":"10.1007\/978-1-4612-0729-0"},{"key":"atypb21","doi-asserted-by":"publisher","DOI":"10.1137\/S1052623497315768"},{"key":"atypb22","doi-asserted-by":"crossref","unstructured":"O. Hernandez-Lerma and J. B. Lasserre,\n                      Further Topics On Discrete-Time Markov Control Processes\n                      , Springer-Verlag. New York, 1999.","DOI":"10.1007\/978-1-4612-0561-6"},{"key":"atypb23","doi-asserted-by":"publisher","DOI":"10.7146\/math.scand.a-11506"},{"key":"atypb24","first-page":"1","volume":"148","author":"L. C.","year":"1983","journal-title":"Math. Centre Tracts"},{"key":"atypb25","doi-asserted-by":"publisher","DOI":"10.1287\/moor.2.1.45"},{"key":"atypb26","doi-asserted-by":"publisher","DOI":"10.1287\/moor.2.3.266"},{"key":"atypb27","unstructured":"J.B. Lasserre,\n                      Moments, Positive Polynomials and Their Applications\n                      , Imperial College Press Optim. Ser. 1, Imperial College Press. London, 2009."},{"key":"atypb28","first-page":"199","volume":"22","author":"Mangasarian O. L.","year":"1997","journal-title":"Acta Math. Vietnam."},{"key":"atypb29","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.6.3.259"},{"key":"atypb30","first-page":"77","volume":"7","author":"Markowitz H.","year":"1952","journal-title":"J. Finance"},{"key":"atypb31","unstructured":"A. Muller and D. Stoyan,\n                      Comparison Methods for Stochastic Models and Risks\n                      , John Wiley and Sons, Chichester, England, 2002."},{"key":"atypb32","doi-asserted-by":"crossref","unstructured":"A. Piunovskiy,\n                      Optimal Control of Random Sequences in Problems with Constraints\n                      , Math. Appl. 410, Kluwer Academic, Dordrecht, 1997.","DOI":"10.1007\/978-94-011-5508-3"},{"key":"atypb33","first-page":"113","author":"Prekopa A.","year":"1970","journal-title":"NJ"},{"key":"atypb34","unstructured":"M. L. Puterman,\n                      Markov Decision Processes: Discrete Stochastic Dynamic Programming\n                      , John Wiley & Sons, Hoboken, NJ, 2005."},{"key":"atypb35","doi-asserted-by":"publisher","DOI":"10.21314\/JOR.2000.038"},{"key":"atypb36","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-010-0393-3"},{"key":"atypb37","doi-asserted-by":"crossref","unstructured":"M. Shaked and J. G. Shanthikumar,\n                      Stochastic Orders\n                      , Springer, New York, 2007.","DOI":"10.1007\/978-0-387-34675-5"},{"key":"atypb38","doi-asserted-by":"publisher","DOI":"10.1287\/opre.42.1.175"},{"key":"atypb39","unstructured":"G. P. Todd,\n                      Mean-Variance Analysis in Portfolio Choice and Capital Markets\n                      , John Wiley & Sons, Hoboken, NJ, 2000."},{"key":"atypb40","first-page":"50","volume":"7","author":"Yang B.","year":"2013","journal-title":"J. Uncertain Systems"}],"container-title":["SIAM Journal on Control and Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/epubs.siam.org\/doi\/pdf\/10.1137\/140969221","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T14:22:41Z","timestamp":1787322161000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/140969221"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1]]},"references-count":40,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2015,1]]}},"alternative-id":["10.1137\/140969221"],"URL":"https:\/\/doi.org\/10.1137\/140969221","relation":{},"ISSN":["0363-0129","1095-7138"],"issn-type":[{"value":"0363-0129","type":"print"},{"value":"1095-7138","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,1]]}}}