{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T03:12:49Z","timestamp":1790651569322,"version":"4.1.0"},"reference-count":43,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2023,2,24]],"date-time":"2023-02-24T00:00:00Z","timestamp":1677196800000},"content-version":"vor","delay-in-days":429,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"LSE New Research Support Fund"},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DMS-1555244"],"award-info":[{"award-number":["DMS-1555244"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DMS-2113637"],"award-info":[{"award-number":["DMS-2113637"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision-making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper was to construct confidence intervals (CIs) for a policy\u2019s value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action-value state function (Q-function) associated with a policy based on series\/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies are conducted to back up our theoretical findings. We apply the proposed method to a dataset from mobile health studies and find that reinforcement learning algorithms could help improve patient\u2019s health status. A Python implementation of the proposed procedure is available at https:\/\/github.com\/shengzhang37\/SAVE.<\/jats:p>","DOI":"10.1111\/rssb.12465","type":"journal-article","created":{"date-parts":[[2021,12,23]],"date-time":"2021-12-23T13:36:23Z","timestamp":1640266583000},"page":"765-793","source":"Crossref","is-referenced-by-count":41,"title":["Statistical Inference of the Value Function for Reinforcement Learning in Infinite-Horizon Settings"],"prefix":"10.1093","volume":"84","author":[{"given":"Chengchun","family":"Shi","sequence":"first","affiliation":[{"name":"London School of Economics and Political Science , London , UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sheng","family":"Zhang","sequence":"additional","affiliation":[{"name":"North Carolina State University , Raleigh , North Carolina , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenbin","family":"Lu","sequence":"additional","affiliation":[{"name":"North Carolina State University , Raleigh , North Carolina , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Song","sequence":"additional","affiliation":[{"name":"North Carolina State University , Raleigh , North Carolina , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,12,22]]},"reference":[{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1214\/009053606000001217","article-title":"Fast learning rates for plug-in classifiers","volume":"35","author":"Audibert","year":"2007","journal-title":"The Annals of Statistics"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1214\/154957805100000104","article-title":"Basic properties of strong mixing conditions. A survey and some open questions","volume":"2","author":"Bradley","year":"2005","journal-title":"Probability Surveys"},{"key":"2023022501102159100_","volume-title":"Openai gym","author":"Brockman","year":"2016"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1016\/j.jeconom.2015.03.010","article-title":"Optimal uniform convergence rates and asymptotic normality for series estimators under weak dependence and weak conditions","volume":"188","author":"Chen","year":"2015","journal-title":"Journal of Econometrics"},{"issue":"4","key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1214\/15-STS527","article-title":"High-dimensional inference: confidence intervals, p-values and R-software hdi","volume":"30","author":"Dezeure","year":"2015","journal-title":"Statistical Science"},{"key":"2023022501102159100_","first-page":"503","article-title":"Tree-based batch mode reinforcement learning","volume":"6","author":"Ernst","year":"2005","journal-title":"Journal of Machine Learning Research"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"963","DOI":"10.1093\/biomet\/asy043","article-title":"Constructing dynamic treatment regimes over indefinite time horizons","volume":"105","author":"Ertefaie","year":"2018","journal-title":"Biometrika"},{"key":"2023022501102159100_","volume-title":"PMLR workshop paper","author":"Fan"},{"key":"2023022501102159100_","volume-title":"Fast rates for the regret of offline reinforcement learning","author":"Hu","year":"2021"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1214\/aos\/1030563984","article-title":"Projection estimation in multiple regression with application to functional ANOVA models","volume":"26","author":"Huang","year":"1998","journal-title":"The Annals of Statistics"},{"key":"2023022501102159100_","first-page":"12519","volume-title":"Advances in neural information processing systems","author":"Janner","year":"2019"},{"key":"2023022501102159100_","volume-title":"Doubly robust off-policy value evaluation for reinforcement learning","author":"Jiang","year":"2016"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","DOI":"10.1145\/3269206.3272021","volume-title":"Real-time bidding with multi-agent reinforcement learning in display advertising","author":"Jin","year":"2018"},{"key":"2023022501102159100_","volume-title":"Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning","author":"Kallus","year":"2019"},{"key":"2023022501102159100_","first-page":"1","article-title":"Double reinforcement learning for efficient off-policy evaluation in markov decision processes","volume":"21","author":"Kallus","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"122","DOI":"10.3390\/robotics2030122","article-title":"Reinforcement learning in robotics: applications and real-world challenges","volume":"2","author":"Kormushev","year":"2013","journal-title":"Robotics"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1080\/01621459.2018.1537919","article-title":"Estimating dynamic treatment regimes in mobile health using V-learning","volume":"115","author":"Luckett","year":"2019","journal-title":"Journal of the American Statistical Association"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1214\/15-AOS1384","article-title":"Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy","volume":"44","author":"Luedtke","year":"2016","journal-title":"The Annals of Statistics"},{"key":"2023022501102159100_","first-page":"719","volume-title":"Toward off-policy learning control with function approximation","author":"Maei","year":"2010"},{"key":"2023022501102159100_","first-page":"60","volume-title":"The OhioT1DM dataset for blood glucose level prediction","author":"Marling","year":"2018"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1111\/1467-9868.00389","article-title":"Optimal dynamic treatment regimes","volume":"65","author":"Murphy","year":"2003","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","DOI":"10.1002\/9780470316887","volume-title":"Markov decision processes: discrete stochastic dynamic programming","author":"Puterman","year":"1994"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"1180","DOI":"10.1214\/10-AOS864","article-title":"Performance guarantees for individualized treatment rules","volume":"39","author":"Qian","year":"2011","journal-title":"The Annals of Statistics"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","DOI":"10.1007\/11564096_32","volume-title":"Neural fitted Q iteration\u2013first experiences with a data efficient neural reinforcement learning method","author":"Riedmiller","year":"2005"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"S-55","DOI":"10.1089\/dia.2008.0132","article-title":"Interpretation of continuous glucose monitoring data: glycemic variability and quality of glycemic control","volume":"11","author":"Rodbard","year":"2009","journal-title":"Diabetes Technology & Therapeutics"},{"key":"2023022501102159100_","volume-title":"Stability results for nonlinear vector autoregressions with an application to a nonlinear error correction model","author":"Saikkonen","year":"2001"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"925","DOI":"10.1214\/17-AOS1570","article-title":"High-dimensional A-learning for optimal dynamic treatment regimes","volume":"46","author":"Shi","year":"2018","journal-title":"The Annals of Statistics"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1111\/rssb.12273","article-title":"Maximin projection learning for optimal treatment decision with heterogeneous individualized treatment effects","volume":"80","author":"Shi","year":"2018","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2023022501102159100_","first-page":"1","article-title":"Breaking the curse of nonregularity with subagging\u2014inference of the mean outcome under optimal treatment regimes","volume":"21","author":"Shi","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2023022501102159100_","volume-title":"Deeply-debiased off-policy interval estimation","author":"Shi","year":"2021"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"2023022501102159100_","volume-title":"Reinforcement learning: an introduction","author":"Sutton","year":"2018"},{"key":"2023022501102159100_","volume-title":"Doubly robust bias reduction in infinite horizon off-policy estimation","author":"Tang","year":"2020"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v29i1.9541","volume-title":"High-confidence off-policy evaluation","author":"Thomas","year":"2015"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"1800","DOI":"10.1093\/ije\/dyv337","article-title":"Cohort profile: the Famingham Heart Study (FHS): overview of milestones in cardiovascular epidemiology","volume":"44","author":"Tsao","year":"2015","journal-title":"International Journal of Epidemiology"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1214\/aos\/1079120131","article-title":"Optimal aggregation of classifiers in statistical learning","volume":"32","author":"Tsybakov","year":"2004","journal-title":"The Annals of Statistics"},{"key":"2023022501102159100_","volume-title":"Minimax weight and Q-function learning for off-policy evaluation","author":"Uehara","year":"2020"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1080\/01621459.2018.1558062","article-title":"Debiased inference on treatment effect in a high-dimensional model","volume":"115","author":"Wang","year":"2020","journal-title":"Journal of the American Statistical Association"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","DOI":"10.1145\/3219819.3219824","volume-title":"Large-scale order dispatch in on-demand ride-hailing platforms: a learning and planning approach","author":"Xu","year":"2018"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1093\/biomet\/ast014","article-title":"Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions","volume":"100","author":"Zhang","year":"2013","journal-title":"Biometrika"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"1541","DOI":"10.1080\/01621459.2017.1345743","article-title":"Estimation of optimal treatment regimes using lists","volume":"113","author":"Zhang","year":"2018","journal-title":"Journal of the American Statistical Association"},{"key":"2023022501102159100_","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1080\/01621459.2014.937488","article-title":"New statistical learning methods for estimating optimal dynamic treatment regimes","volume":"110","author":"Zhao","year":"2015","journal-title":"Journal of the American Statistical Association"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/rssb.12465","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/rssb.12465","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/am-pdf\/10.1111\/rssb.12465","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/3\/765\/49322491\/jrsssb_84_3_765.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/3\/765\/49322491\/jrsssb_84_3_765.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T20:21:08Z","timestamp":1699993268000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/84\/3\/765\/7056157"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,22]]},"references-count":43,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,12,22]]},"published-print":{"date-parts":[[2022,7,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12465","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,7]]},"published":{"date-parts":[[2021,12,22]]}}}