{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T17:08:15Z","timestamp":1788541695708,"version":"build-2803163510"},"reference-count":57,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,12]]},"abstract":"<jats:title>Summary<\/jats:title>\n               <jats:p>We study the efficient off-policy evaluation of natural stochastic policies, which are defined in terms of deviations from the unknown behaviour policy. This is a departure from the literature on off-policy evaluation that largely considers the evaluation of explicitly specified policies. Crucially, off-line reinforcement learning with natural stochastic policies can help alleviate issues of weak overlap, lead to policies that build upon current practice and improve policies\u2019 implementability in practice. Compared with the classic case of a prespecified evaluation policy, when evaluating natural stochastic policies, the efficiency bound, which measures the best-achievable estimation error, is inflated since the evaluation policy itself is unknown. In this paper we derive the efficiency bounds of two major types of natural stochastic policies: tilting policies and modified treatment policies. We then propose efficient nonparametric estimators that attain the efficiency bounds under lax conditions and enjoy a partial double robustness property.<\/jats:p>","DOI":"10.1093\/biomet\/asad059","type":"journal-article","created":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T23:40:47Z","timestamp":1695858047000},"page":"51-69","source":"Crossref","is-referenced-by-count":5,"title":["Efficient evaluation of natural stochastic policies in off-line reinforcement learning"],"prefix":"10.1093","volume":"111","author":[{"given":"Nathan","family":"Kallus","sequence":"first","affiliation":[{"name":"Department of Operations Research and Information Engineering, Cornell University , 2 West Loop Road , New York, New York 10044, U.S.A"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9017-3105","authenticated-orcid":false,"given":"Masatoshi","family":"Uehara","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Cornell University , 2 West Loop Road , New York, New York 10044, U.S.A"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"2024021219291574700_asad059-B1","first-page":"1","article-title":"On the theory of policy gradient methods: optimality, approximation, and distribution shift","volume":"22","author":"Agarwal","year":"2021","journal-title":"J. Mach. Learn. Res"},{"key":"2024021219291574700_asad059-B2","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10994-007-5038-2","article-title":"Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path","volume":"71","author":"Antos","year":"2008","journal-title":"Mach. Learn"},{"key":"2024021219291574700_asad059-B3","first-page":"654","volume-title":"Proc. 36th Int. Conf. Mach. Learn","author":"Bibaut","year":"2019"},{"key":"2024021219291574700_asad059-B4","doi-asserted-by":"crossref","first-page":"1112","DOI":"10.1080\/01621459.2017.1305274","article-title":"Assessing time-varying causal effect moderation in mobile health","volume":"113","author":"Boruvka","year":"2018","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B5","author":"Brockman","year":"2016"},{"key":"2024021219291574700_asad059-B6","doi-asserted-by":"crossref","first-page":"C1","DOI":"10.1111\/ectj.12097","article-title":"Double\/debiased machine learning for treatment and structural parameters","volume":"21","author":"Chernozhukov","year":"2018","journal-title":"Economet. J"},{"key":"2024021219291574700_asad059-B7","author":"Chernozhukov","year":"2019"},{"key":"2024021219291574700_asad059-B8","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1111\/rssb.12362","article-title":"Causal mediation analysis for stochastic interventions","volume":"82","author":"D\u00edaz","year":"2020","journal-title":"J. R. Statist. Soc. B"},{"key":"2024021219291574700_asad059-B9","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1515\/ijb-2013-0014","article-title":"Assessing the causal effect of policies: an example using stochastic interventions","volume":"9","author":"D\u00edaz","year":"2013","journal-title":"Int. J. Biostatist"},{"key":"2024021219291574700_asad059-B10","first-page":"219","volume-title":"Targeted Learning in Data Science","author":"D\u00edaz","year":"2018"},{"key":"2024021219291574700_asad059-B11","author":"D\u00edaz","year":"2021"},{"key":"2024021219291574700_asad059-B12","first-page":"227","article-title":"Hierarchical reinforcement learning with the MAXQ value function decomposition","volume":"13","author":"Dietterich","year":"2000","journal-title":"J. Artif. Intel. Res"},{"key":"2024021219291574700_asad059-B13","first-page":"2701","volume-title":"Proc. 37th Int. Conf. Mach. Learn","author":"Duan","year":"2020"},{"key":"2024021219291574700_asad059-B14","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1214\/14-STS500","article-title":"Doubly robust policy evaluation and optimization","volume":"29","author":"Dudik","year":"2014","journal-title":"Statist. Sci."},{"key":"2024021219291574700_asad059-B15","first-page":"503","article-title":"Tree-based batch mode reinforcement learning","volume":"6","author":"Ernst","year":"2005","journal-title":"J. Mach. Learn. Res"},{"key":"2024021219291574700_asad059-B16","doi-asserted-by":"crossref","first-page":"963","DOI":"10.1093\/biomet\/asy043","article-title":"Constructing dynamic treatment regimes over indefinite time horizons","volume":"105","author":"Ertefaie","year":"2018","journal-title":"Biometrika"},{"key":"2024021219291574700_asad059-B17","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1038\/s41591-018-0310-5","article-title":"Guidelines for reinforcement learning in healthcare","volume":"25","author":"Gottesman","year":"2019","journal-title":"Nature Med"},{"key":"2024021219291574700_asad059-B18","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1001\/jamapsychiatry.2013.4642","article-title":"A smartphone application to support recovery from alcoholism: a randomized clinical trial","volume":"71","author":"Gustafson","year":"2014","journal-title":"JAMA Psychiatry"},{"key":"2024021219291574700_asad059-B19","doi-asserted-by":"crossref","first-page":"315","DOI":"10.2307\/2998560","article-title":"On the role of the propensity score in efficient semiparametric estimation of average treatment effects","volume":"66","author":"Hahn","year":"1998","journal-title":"Econometrica"},{"key":"2024021219291574700_asad059-B20","doi-asserted-by":"crossref","first-page":"5260","DOI":"10.1002\/sim.5907","article-title":"Estimation of the effect of interventions that modify the received treatment","volume":"32","author":"Haneuse","year":"2013","journal-title":"Statist. Med"},{"key":"2024021219291574700_asad059-B21","first-page":"652","volume-title":"Proc. 33rd Int. Conf. Mach. Learn","author":"Jiang","year":"2016"},{"key":"2024021219291574700_asad059-B22","first-page":"3325","volume-title":"Proc. 33rd Int. Conf. Neural Info. Proces. Syst","author":"Kallus","year":"2019"},{"key":"2024021219291574700_asad059-B23","first-page":"1","article-title":"Double reinforcement learning for efficient off-policy evaluation in Markov decision processes","volume":"21","author":"Kallus","year":"2020","journal-title":"J. Mach. Learn. Res"},{"key":"2024021219291574700_asad059-B24","author":"Kallus","year":"2023"},{"key":"2024021219291574700_asad059-B25","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1080\/01621459.2017.1422737","article-title":"Nonparametric causal effects based on incremental propensity score interventions","volume":"114","author":"Kennedy","year":"2019","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B26","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611974188","volume-title":"Adaptive Treatment Strategies in Practice: Planning Trials and Analyzing Data for Personalized Medicine","author":"Kosorok","year":"2015"},{"key":"2024021219291574700_asad059-B27","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1111\/1467-9868.00185","article-title":"Semiparametric methods for response-selective and missing data problems in regression","volume":"61","author":"Lawless","year":"1999","journal-title":"J. R. Statist. Soc. B"},{"key":"2024021219291574700_asad059-B28","first-page":"1","volume-title":"Proc. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies","author":"Liao","year":"2020"},{"key":"2024021219291574700_asad059-B29","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1080\/01621459.2020.1807993","article-title":"Off-policy estimation of long-term average outcomes with applications to mobile health","volume":"116","author":"Liao","year":"2021","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B30","first-page":"5361","volume-title":"Proc. 32nd Int. Conf. Neural Info. Proces. Syst","author":"Liu","year":"2018"},{"key":"2024021219291574700_asad059-B31","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1080\/01621459.2018.1537919","article-title":"Estimating dynamic treatment regimes in mobile health using v-learning","volume":"115","author":"Luckett","year":"2020","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B32","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1111\/1467-9868.00389","article-title":"Optimal dynamic treatment regimes","volume":"65","author":"Murphy","year":"2003","journal-title":"J. R. Statist. Soc. B"},{"key":"2024021219291574700_asad059-B33","doi-asserted-by":"crossref","first-page":"1410","DOI":"10.1198\/016214501753382327","article-title":"Marginal mean models for dynamic regimes","volume":"96","author":"Murphy","year":"2001","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B34","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1111\/j.1541-0420.2011.01685.x","article-title":"Population intervention causal effects based on stochastic interventions","volume":"68","author":"Mu\u00f1oz","year":"2012","journal-title":"Biometrics"},{"key":"2024021219291574700_asad059-B35","first-page":"2318","volume-title":"Proc. 33rd Int. Conf. Neural Info. Proces. Syst","author":"Nachum","year":"2019"},{"key":"2024021219291574700_asad059-B36","first-page":"4634","volume-title":"Proc. 33rd AAAI Conf. Artif. Intel","author":"Narita","year":"2019"},{"key":"2024021219291574700_asad059-B37","first-page":"759","volume-title":"Proc. 17th Int. Conf. Mach. Learn","author":"Precup","year":"2000"},{"key":"2024021219291574700_asad059-B38","doi-asserted-by":"crossref","first-page":"245","DOI":"10.3200\/JACH.57.2.245-248","article-title":"Internet and mobile phone text messaging intervention for college smokers","volume":"57","author":"Riley","year":"2008","journal-title":"J. Am. Coll. Health"},{"key":"2024021219291574700_asad059-B39","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/978-1-4419-9076-1_11","volume-title":"Proceedings of the Second Seattle Symposium in Biostatistics","author":"Robins","year":"2004"},{"key":"2024021219291574700_asad059-B40","first-page":"1","volume-title":"Statistical Models in Epidemiology, the Environment, and Clinical Trials","author":"Robins","year":"2000"},{"key":"2024021219291574700_asad059-B41","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1080\/01621459.1994.10476818","article-title":"Estimation of regression coefficients when some regressors are not always observed","volume":"89","author":"Robins","year":"1994","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B42","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1093\/biomet\/70.1.41","article-title":"The central role of the propensity score in observational studies for causal effects","volume":"70","author":"Rosenbaum","year":"1983","journal-title":"Biometrika"},{"key":"2024021219291574700_asad059-B43","first-page":"1889","volume-title":"Proc. 32nd Int. Conf. Mach. Learn","author":"Schulman","year":"2015"},{"key":"2024021219291574700_asad059-B44","author":"Shi","year":"2021"},{"key":"2024021219291574700_asad059-B45","first-page":"514","volume-title":"Proc. 25th Conf. Uncertainty in Artif. Intel","author":"Shpitser","year":"2009"},{"key":"2024021219291574700_asad059-B46","first-page":"1","author":"Tang","year":"2020"},{"key":"2024021219291574700_asad059-B47","author":"Tchetgen Tchetgen","year":"2013"},{"key":"2024021219291574700_asad059-B48","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1037\/hea0000304","article-title":"Behavioral response to a just-in-time adaptive intervention (JITAI) to reduce sedentary behavior in obese adults: Implications for JITAI optimization","volume":"34","author":"Thomas","year":"2015","journal-title":"Health Psychol"},{"key":"2024021219291574700_asad059-B49","first-page":"2139","volume-title":"Proc. 33rd Int. Conf. Mach. Learn","author":"Thomas","year":"2016"},{"key":"2024021219291574700_asad059-B50","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-21700-0","volume-title":"Unified Methods for Censored Longitudinal Data and Causality","author":"van der Laan","year":"2003"},{"key":"2024021219291574700_asad059-B51","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511802256","volume-title":"Asymptotic Statistics","author":"van der Vaart","year":"1998"},{"key":"2024021219291574700_asad059-B52","author":"Wang","year":"2020"},{"key":"2024021219291574700_asad059-B53","first-page":"3948","volume-title":"Proc. 23rd Int. Conf. Artif. Intel. Statist","author":"Yin","year":"2020"},{"key":"2024021219291574700_asad059-B54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1515\/em-2012-0001","article-title":"Identification, estimation and approximation of risk under interventions that depend on the natural value of treatment using observational data","volume":"3","author":"Young","year":"2014","journal-title":"Epidemiol. Meth"},{"key":"2024021219291574700_asad059-B55","doi-asserted-by":"crossref","first-page":"938","DOI":"10.1080\/01621459.2018.1469993","article-title":"Inverse probability weighted estimation of risk under representative interventions in observational studies","volume":"114","author":"Young","year":"2019","journal-title":"J. Am. Statist. Assoc"},{"key":"2024021219291574700_asad059-B56","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1093\/biomet\/ast014","article-title":"Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions","volume":"100","author":"Zhang","year":"2013","journal-title":"Biometrika"},{"key":"2024021219291574700_asad059-B57","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1007\/978-1-4419-9782-1_27","volume-title":"Targeted Learning","author":"Zheng","year":"2011"}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/biomet\/advance-article-pdf\/doi\/10.1093\/biomet\/asad059\/55452458\/asad059.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/111\/1\/51\/56665912\/asad059.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/111\/1\/51\/56665912\/asad059.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,12]],"date-time":"2024-02-12T19:30:38Z","timestamp":1707766238000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/111\/1\/51\/7284104"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":57,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,9,27]]},"published-print":{"date-parts":[[2024,2,12]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asad059","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,3,1]]},"published":{"date-parts":[[2023,9,27]]}}}