{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:59:29Z","timestamp":1791392369975,"version":"4.3.3"},"reference-count":97,"publisher":"Institute of Mathematical Statistics","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Statist."],"published-print":{"date-parts":[[2025,8,1]]},"abstract":"<jats:p>This paper studies offline policy learning, which aims at utilizing observations collected a priori (from either fixed or adaptively evolving behavior policies) to learn an optimal individualized decision rule that achieves the best overall outcomes for a given population. Existing policy learning methods rely on a uniform overlap assumption, that is, the propensities of exploring all actions for all individual characteristics must be lower bounded; the performance of the existing methods thus depends on the worst-case propensity in the offline data set. As one has no control over the data collection process, this assumption can be unrealistic in many situations, especially when the behavior policies are allowed to evolve over time with diminishing propensities. In this paper, we propose Pessimistic Policy Learning (PPL), a new algorithm that optimizes lower confidence bounds (LCBs)\u2014instead of point estimates\u2014of the policy values. The LCBs are constructed using knowledge of the behavior policies for collecting the offline data. Without assuming any uniform overlap condition, we establish a data-dependent upper bound for the suboptimality of our algorithm, which only depends on (i) the overlap for the optimal policy, and (ii) the complexity of the policy class we optimize over. As an implication, for adaptively collected data, we ensure efficient policy learning as long as the propensities for optimal actions are lower bounded over time, while those for suboptimal ones are allowed to diminish arbitrarily fast. In our theoretical analysis, we develop a new self-normalized type concentration inequality for inverse-propensity-weighting estimators, generalizing the well-known empirical Bernstein\u2019s inequality to unbounded and non-i.i.d. data. We complement our theory with an efficient optimization algorithm via Majorization\u2013Minimization and policy tree search, as well as extensive simulation studies and real-world applications that demonstrate the efficacy of PPL.<\/jats:p>","DOI":"10.1214\/25-aos2511","type":"journal-article","created":{"date-parts":[[2025,9,12]],"date-time":"2025-09-12T05:22:51Z","timestamp":1757654571000},"page":"1483-1512","source":"Crossref","is-referenced-by-count":1,"title":["Policy learning \u201cwithout\u201d overlap: Pessimism and generalized empirical Bernstein\u2019s inequality"],"prefix":"10.1214","volume":"53","author":[{"given":"Ying","family":"Jin","sequence":"first","affiliation":[{"name":"Data Science Initiative and Department of Health Care Policy, Harvard University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhimei","family":"Ren","sequence":"additional","affiliation":[{"name":"Department of Statistics and Data Science, University of Pennsylvania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhuoran","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Statistics and Data Science, Yale University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhaoran","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering and Management Sciences, Northwestern University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"90","doi-asserted-by":"publisher","unstructured":"<small>Zhang, B.<\/small>, <small>Tsiatis, A. A.<\/small>, <small>Davidian, M.<\/small>, <small>Zhang, M.<\/small> and <small>Laber, E.<\/small> (2012). Estimating optimal treatment regimes from a classification perspective. <i>Stat<\/i> <b>1<\/b> 103\u2013114.","DOI":"10.1002\/sta.411"},{"key":"93","doi-asserted-by":"publisher","unstructured":"<small>Zhao, Y.<\/small>, <small>Zeng, D.<\/small>, <small>Rush, A. J.<\/small> and <small>Kosorok, M. R.<\/small> (2012). Estimating individualized treatment rules using outcome weighted learning. <i>J. Amer. Statist. Assoc.<\/i> <b>107<\/b> 1106\u20131118.","DOI":"10.1080\/01621459.2012.695674"},{"key":"62","doi-asserted-by":"publisher","unstructured":"<small>Schick, A.<\/small> (1986). On asymptotically efficient estimation in semiparametric models. <i>Ann. Statist.<\/i> <b>14<\/b> 1139\u20131151.","DOI":"10.1214\/aos\/1176350055"},{"key":"47","unstructured":"<small>LIU, Q.<\/small>, <small>LI, L.<\/small>, <small>TANG, Z.<\/small> and <small>ZHOU, D.<\/small> (2018). Breaking the curse of horizon: Infinite-horizon off-policy estimation. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>31<\/b>."},{"key":"13","doi-asserted-by":"crossref","unstructured":"<small>CHAKRABORTY, B.<\/small> and <small>MURPHY, S. A.<\/small> (2014). Dynamic treatment regimes. <i>Annu. Rev. Stat. Appl.<\/i> <b>1<\/b> 447\u2013464.","DOI":"10.1146\/annurev-statistics-022513-115553"},{"key":"96","doi-asserted-by":"publisher","unstructured":"<small>ZHOU, Z.<\/small>, <small>ATHEY, S.<\/small> and <small>WAGER, S.<\/small> (2023). Offline multi-action policy learning: Generalization and optimization. <i>Oper. Res.<\/i> <b>71<\/b> 148\u2013183.","DOI":"10.1287\/opre.2022.2271"},{"key":"39","doi-asserted-by":"publisher","unstructured":"<small>KITAGAWA, T.<\/small> and <small>TETENOV, A.<\/small> (2018). Who should be treated? Empirical welfare maximization methods for treatment choice. <i>Econometrica<\/i> <b>86<\/b> 591\u2013616.","DOI":"10.3982\/ECTA13288"},{"key":"70","unstructured":"<small>SWAMINATHAN, A.<\/small> and <small>JOACHIMS, T.<\/small> (2015). Batch learning from logged bandit feedback through counterfactual risk minimization. <i>J. Mach. Learn. Res.<\/i> <b>16<\/b> 1731\u20131755."},{"key":"73","unstructured":"<small>UEHARA, M.<\/small>, <small>HUANG, J.<\/small> and <small>JIANG, N.<\/small> (2020). Minimax weight and q-function learning for off-policy evaluation. In <i>International Conference on Machine Learning<\/i> 9659\u20139668. PMLR."},{"key":"41","doi-asserted-by":"publisher","unstructured":"<small>Koltchinskii, V.<\/small> (2006). Local Rademacher complexities and oracle inequalities in risk minimization. <i>Ann. Statist.<\/i> <b>34<\/b> 2593\u20132656.","DOI":"10.1214\/009053606000001019"},{"key":"4","doi-asserted-by":"publisher","unstructured":"<small>ATHEY, S.<\/small> and <small>WAGER, S.<\/small> (2021). Policy learning with observational data. <i>Econometrica<\/i> <b>89<\/b> 133\u2013161.","DOI":"10.3982\/ecta15732"},{"key":"6","doi-asserted-by":"publisher","unstructured":"<small>BARTLETT, P. L.<\/small>, <small>JORDAN, M. I.<\/small> and <small>MCAULIFFE, J. D.<\/small> (2006). Convexity, classification, and risk bounds. <i>J. Amer. Statist. Assoc.<\/i> <b>101<\/b> 138\u2013156.","DOI":"10.1198\/016214505000000907"},{"key":"95","doi-asserted-by":"publisher","unstructured":"<small>ZHOU, X.<\/small>, <small>MAYER-HAMBLETT, N.<\/small>, <small>KHAN, U.<\/small> and <small>KOSOROK, M. R.<\/small> (2017). Residual weighted learning for estimating individualized treatment rules. <i>J. Amer. Statist. Assoc.<\/i> <b>112<\/b> 169\u2013187.","DOI":"10.1080\/01621459.2015.1093947"},{"key":"2","doi-asserted-by":"crossref","unstructured":"<small>ANSCOMBE, F. J.<\/small> (1963). Sequential medical trials. <i>J. Amer. Statist. Assoc.<\/i> <b>58<\/b> 365\u2013383.","DOI":"10.1080\/01621459.1963.10500851"},{"key":"27","doi-asserted-by":"publisher","unstructured":"<small>HADAD, V.<\/small>, <small>HIRSHBERG, D. A.<\/small>, <small>ZHAN, R.<\/small>, <small>WAGER, S.<\/small> and <small>ATHEY, S.<\/small> (2021). Confidence intervals for policy evaluation in adaptive experiments. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>118<\/b> Paper No. e2014602118, 10.","DOI":"10.1073\/pnas.2014602118"},{"key":"72","doi-asserted-by":"crossref","unstructured":"<small>THOMPSON, W. R.<\/small> (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. <i>Biometrika<\/i> <b>25<\/b> 285\u2013294.","DOI":"10.1093\/biomet\/25.3-4.285"},{"key":"32","doi-asserted-by":"crossref","unstructured":"<small>JIN, Y.<\/small>, <small>REN, Z.<\/small> and <small>CAND\u00c8S, E. J.<\/small> (2023). Sensitivity analysis of individual treatment effects: A robust conformal inference approach. <i>Proc. Natl. Acad. Sci. USA<\/i> <b>120<\/b> Paper No. e2214889120, 13.","DOI":"10.1073\/pnas.2214889120"},{"key":"30","doi-asserted-by":"publisher","unstructured":"<small>Imbens, G. W.<\/small> and <small>Rubin, D. B.<\/small> (2015). <i>Causal Inference\u2014for Statistics, Social, and Biomedical Sciences<\/i>: <i>An Introduction<\/i>. Cambridge Univ. Press, New York.","DOI":"10.1017\/CBO9781139025751"},{"key":"24","doi-asserted-by":"publisher","unstructured":"<small>Fan, J.<\/small> and <small>Li, R.<\/small> (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. <i>J. Amer. Statist. Assoc.<\/i> <b>96<\/b> 1348\u20131360.","DOI":"10.1198\/016214501753382273"},{"key":"17","doi-asserted-by":"crossref","unstructured":"<small>Collins, L. M.<\/small>, <small>Murphy, S. A.<\/small> and <small>Strecher, V.<\/small> (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent eHealth interventions. <i>Am. J. Prev. Med.<\/i> <b>32<\/b> S112\u2013S118.","DOI":"10.1016\/j.amepre.2007.01.022"},{"key":"53","doi-asserted-by":"publisher","unstructured":"<small>Murphy, S. A.<\/small> (2005). An experimental design for the development of adaptive treatment strategies. <i>Stat. Med.<\/i> <b>24<\/b> 1455\u20131481.","DOI":"10.1002\/sim.2022"},{"key":"97","doi-asserted-by":"publisher","unstructured":"<small>Zou, H.<\/small> (2006). The adaptive lasso and its oracle properties. <i>J. Amer. Statist. Assoc.<\/i> <b>101<\/b> 1418\u20131429.","DOI":"10.1198\/016214506000000735"},{"key":"22","doi-asserted-by":"publisher","unstructured":"<small>Donoho, D. L.<\/small> and <small>Johnstone, I. M.<\/small> (1994). Ideal spatial adaptation by wavelet shrinkage. <i>Biometrika<\/i> <b>81<\/b> 425\u2013455.","DOI":"10.1093\/biomet\/81.3.425"},{"key":"43","doi-asserted-by":"publisher","unstructured":"<small>Lai, T. L.<\/small> and <small>Robbins, H.<\/small> (1985). Asymptotically efficient adaptive allocation rules. <i>Adv. in Appl. Math.<\/i> <b>6<\/b> 4\u201322.","DOI":"10.1016\/0196-8858(85)90002-8"},{"key":"77","doi-asserted-by":"crossref","unstructured":"<small>Vapnik, V. N.<\/small> and <small>Chervonenkis, A. Y.<\/small> (2015). On the uniform convergence of relative frequencies of events to their probabilities. In <i>Measures of Complexity<\/i> 11\u201330. Springer, Cham. Reprint of Theor. Probability Appl. <b>1<\/b>(6) (1971), 264\u2013280.","DOI":"10.1007\/978-3-319-21852-6_3"},{"key":"19","doi-asserted-by":"publisher","unstructured":"<small>de la Pe\u00f1a, V. H.<\/small>, <small>Klass, M. J.<\/small> and <small>Lai, T. L.<\/small> (2004). Self-normalized processes: Exponential inequalities, moment bounds and iterated logarithm laws. <i>Ann. Probab.<\/i> <b>32<\/b> 1902\u20131933.","DOI":"10.1214\/009117904000000397"},{"key":"20","doi-asserted-by":"publisher","unstructured":"<small>de la Pe\u00f1a, V. H.<\/small>, <small>Lai, T. L.<\/small> and <small>Shao, Q.-M.<\/small> (2009). <i>Self-Normalized Processes<\/i>: <i>Limit Theory and Statistical Applications<\/i>. <i>Probability and Its Applications<\/i> (<i>New York<\/i>). Springer, Berlin.","DOI":"10.1007\/978-3-540-85636-8"},{"key":"5","doi-asserted-by":"publisher","unstructured":"<small>Bartlett, P. L.<\/small>, <small>Bousquet, O.<\/small> and <small>Mendelson, S.<\/small> (2005). Local Rademacher complexities. <i>Ann. Statist.<\/i> <b>33<\/b> 1497\u20131537.","DOI":"10.1214\/009053605000000282"},{"key":"16","doi-asserted-by":"publisher","unstructured":"<small>Chernozhukov, V.<\/small>, <small>Chetverikov, D.<\/small>, <small>Demirer, M.<\/small>, <small>Duflo, E.<\/small>, <small>Hansen, C.<\/small>, <small>Newey, W.<\/small> and <small>Robins, J.<\/small> (2018). Double\/debiased machine learning for treatment and structural parameters. <i>Econom. J.<\/i> <b>21<\/b> C1\u2013C68.","DOI":"10.1111\/ectj.12097"},{"key":"28","doi-asserted-by":"publisher","unstructured":"<small>Hastie, T.<\/small>, <small>Tibshirani, R.<\/small> and <small>Friedman, J.<\/small> (2009). <i>The Elements of Statistical Learning<\/i>: <i>Data Mining, Inference, and Prediction<\/i>, 2nd ed. <i>Springer Series in Statistics<\/i>. Springer, New York.","DOI":"10.1007\/978-0-387-84858-7"},{"key":"60","doi-asserted-by":"crossref","unstructured":"<small>Robins, J. M.<\/small>, <small>Rotnitzky, A.<\/small> and <small>Zhao, L. P.<\/small> (1994). Estimation of regression coefficients when some regressors are not always observed. <i>J. Amer. Statist. Assoc.<\/i> <b>89<\/b> 846\u2013866.","DOI":"10.1080\/01621459.1994.10476818"},{"key":"78","doi-asserted-by":"publisher","unstructured":"<small>WAUDBY-SMITH, I.<\/small> and <small>RAMDAS, A.<\/small> (2024). Estimating means of bounded random variables by betting. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>86<\/b> 1\u201327.","DOI":"10.1093\/jrsssb\/qkad009"},{"key":"3","doi-asserted-by":"crossref","unstructured":"<small>ATHEY, S.<\/small> and <small>IMBENS, G. W.<\/small> (2017). The econometrics of randomized experiments. In <i>Handbook of Economic Field Experiments<\/i> <b>1<\/b> 73\u2013140. Elsevier, Amsterdam.","DOI":"10.1016\/bs.hefe.2016.10.003"},{"key":"7","unstructured":"<small>BEMBOM, O.<\/small> and <small>VAN DER LAAN, M. J.<\/small> (2008). Data-adaptive selection of the truncation level for inverse-probability-of-treatment-weighted estimators."},{"key":"8","doi-asserted-by":"crossref","unstructured":"<small>BERTSIMAS, D.<\/small>, <small>KALLUS, N.<\/small>, <small>WEINSTEIN, A. M.<\/small> and <small>ZHUO, Y. D.<\/small> (2017). Personalized diabetes management using electronic medical records. <i>Diabetes Care<\/i> <b>40<\/b> 210\u2013217.","DOI":"10.2337\/dc16-0826"},{"key":"9","unstructured":"<small>BIBAUT, A.<\/small>, <small>KALLUS, N.<\/small>, <small>DIMAKOPOULOU, M.<\/small>, <small>CHAMBAZ, A.<\/small> and <small>VAN DER LAAN, M.<\/small> (2021). Risk minimization from adaptively collected data: Guarantees for supervised and policy learning. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 19261\u201319273."},{"key":"10","unstructured":"<small>BOTTOU, L.<\/small>, <small>PETERS, J.<\/small>, <small>QUI\u00d1ONERO-CANDELA, J.<\/small>, <small>CHARLES, D. X.<\/small>, <small>CHICKERING, D. M.<\/small>, <small>PORTUGALY, E.<\/small>, <small>RAY, D.<\/small>, <small>SIMARD, P.<\/small> and <small>SNELSON, E.<\/small> (2013). Counterfactual reasoning and learning systems: The example of computational advertising. <i>J. Mach. Learn. Res.<\/i> <b>14<\/b> 3207\u20133260."},{"key":"11","unstructured":"<small>BRANSON, Z.<\/small>, <small>KENNEDY, E. H.<\/small>, <small>BALAKRISHNAN, S.<\/small> and <small>WASSERMAN, L.<\/small> (2023). Causal effect estimation after propensity score trimming with continuous treatments. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2309.00706\">2309.00706<\/a>."},{"key":"12","unstructured":"<small>BUCKMAN, J.<\/small>, <small>GELADA, C.<\/small> and <small>BELLEMARE, M. G.<\/small> (2020). The importance of pessimism in fixed-dataset policy optimization. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2009.06799\">2009.06799<\/a>."},{"key":"14","unstructured":"<small>CHEN, J.<\/small> and <small>JIANG, N.<\/small> (2022). Offline reinforcement learning under value and density-ratio realizability: The power of gaps. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2203.13935\">2203.13935<\/a>."},{"key":"15","unstructured":"<small>CHEN, X.<\/small>, <small>QI, Z.<\/small> and <small>WAN, R.<\/small> (2023). STEEL: Singularity-aware reinforcement learning. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2301.13152\">2301.13152<\/a>."},{"key":"18","unstructured":"<small>DANIELY, A.<\/small>, <small>SABATO, S.<\/small>, <small>BEN-DAVID, S.<\/small> and <small>SHALEV-SHWARTZ, S.<\/small> (2011). Multiclass learnability and the ERM principle. In <i>Proceedings of the<\/i> 24<i>th Annual Conference on Learning Theory<\/i> 207\u2013232. JMLR Workshop and Conference Proceedings."},{"key":"21","unstructured":"<small>DIMAKOPOULOU, M.<\/small>, <small>ZHOU, Z.<\/small>, <small>ATHEY, S.<\/small> and <small>IMBENS, G.<\/small> (2017). Estimation considerations in contextual bandits. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/1711.07077\">1711.07077<\/a>."},{"key":"23","unstructured":"<small>DUCHI, J.<\/small> and <small>NAMKOONG, H.<\/small> (2019). Variance-based regularization with convex objectives. <i>J. Mach. Learn. Res.<\/i> <b>20<\/b> Paper No. 68, 55."},{"key":"25","doi-asserted-by":"crossref","unstructured":"<small>FARIAS, V. F.<\/small> and <small>LI, A. A.<\/small> (2019). Learning preferences with side information. <i>Manage. Sci.<\/i> <b>65<\/b> 3131\u20133149.","DOI":"10.1287\/mnsc.2018.3092"},{"key":"26","doi-asserted-by":"publisher","unstructured":"<small>GUI, Y.<\/small>, <small>HORE, R.<\/small>, <small>REN, Z.<\/small> and <small>BARBER, R. F.<\/small> (2024). Conformalized survival analysis with adaptive cut-offs. <i>Biometrika<\/i> <b>111<\/b> 459\u2013477.","DOI":"10.1093\/biomet\/asad076"},{"key":"29","doi-asserted-by":"publisher","unstructured":"<small>HIRANO, K.<\/small> and <small>PORTER, J. R.<\/small> (2009). Asymptotics for statistical treatment rules. <i>Econometrica<\/i> <b>77<\/b> 1683\u20131701.","DOI":"10.3982\/ECTA6630"},{"key":"31","doi-asserted-by":"crossref","unstructured":"<small>JIN, Y.<\/small> (2023). Upper bounds on the Natarajan dimensions of some function classes. In 2023 <i>IEEE International Symposium on Information Theory<\/i> (<i>ISIT<\/i>) 1020\u20131025. IEEE Comput. Soc., Los Alamitos.","DOI":"10.1109\/ISIT54713.2023.10206618"},{"key":"33","doi-asserted-by":"crossref","unstructured":"<small>JIN, Y.<\/small>, <small>REN, Z.<\/small>, <small>YANG, Z.<\/small> and <small>WANG, Z.<\/small> (2025). Supplement to \u201cPolicy learning \u201cwithout\u201d overlap: Pessimism and generalized empirical Bernstein\u2019s inequality.\u201d <a href=\"https:\/\/doi.org\/10.1214\/25-AOS2511SUPP\">https:\/\/doi.org\/10.1214\/25-AOS2511SUPP<\/a>","DOI":"10.1214\/25-AOS2511SUPP"},{"key":"34","unstructured":"<small>JIN, Y.<\/small>, <small>YANG, Z.<\/small> and <small>WANG, Z.<\/small> (2021). Is pessimism provably efficient for offline rl? In <i>International Conference on Machine Learning<\/i> 5084\u20135096. PMLR."},{"key":"36","doi-asserted-by":"publisher","unstructured":"<small>KENNEDY, E. H.<\/small> (2019). Nonparametric causal effects based on incremental propensity score interventions. <i>J. Amer. Statist. Assoc.<\/i> <b>114<\/b> 645\u2013656.","DOI":"10.1080\/01621459.2017.1422737"},{"key":"37","unstructured":"<small>KHAN, S.<\/small>, <small>SAVESKI, M.<\/small> and <small>UGANDER, J.<\/small> (2023). Off-policy evaluation beyond overlap: Partial identification through smoothness. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2305.11812\">2305.11812<\/a>."},{"key":"38","unstructured":"<small>KIM, E. S.<\/small>, <small>HERBST, R. S.<\/small>, <small>WISTUBA, I. I.<\/small>, <small>LEE, J. J.<\/small>, <small>BLUMENSCHEIN, G. R.<\/small>, <small>TSAO, A.<\/small>, <small>STEWART, D. J.<\/small>, <small>HICKS, M. E.<\/small>, <small>ERASMUS, J.<\/small> et al. (2011). The BATTLE trial: Personalizing therapy for lung CancerThe BATTLE trial: Personalizing therapy for lung cancer. <i>Cancer Discov.<\/i> <b>1<\/b> 44\u201353."},{"key":"40","doi-asserted-by":"crossref","unstructured":"<small>KOHAVI, R.<\/small>, <small>TANG, D.<\/small> and <small>XU, Y.<\/small> (2020). <i>Trustworthy Online Controlled Experiments<\/i>: <i>A Practical Guide to a\/b Testing<\/i>. Cambridge Univ. Press, Cambridge.","DOI":"10.1017\/9781108653985"},{"key":"42","unstructured":"<small>KUZBORSKIJ, I.<\/small>, <small>VERNADE, C.<\/small>, <small>GYORGY, A.<\/small> and <small>SZEPESV\u00c1RI, C.<\/small> (2021). Confident off-policy evaluation and selection through self-normalized importance weighting. In <i>International Conference on Artificial Intelligence and Statistics<\/i> 640\u2013648. PMLR."},{"key":"44","unstructured":"<small>LEE, J.<\/small>, <small>JEON, W.<\/small>, <small>LEE, B.<\/small>, <small>PINEAU, J.<\/small> and <small>KIM, K.-E.<\/small> (2021). Optidice: Offline policy optimization via stationary distribution correction estimation. In <i>International Conference on Machine Learning<\/i> 6120\u20136130. PMLR."},{"key":"46","doi-asserted-by":"crossref","unstructured":"<small>LI, L.<\/small>, <small>CHU, W.<\/small>, <small>LANGFORD, J.<\/small> and <small>WANG, X.<\/small> (2011). Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In <i>Proceedings of the Fourth ACM International Conference on Web Search and Data Mining<\/i> 297\u2013306.","DOI":"10.1145\/1935826.1935878"},{"key":"48","doi-asserted-by":"publisher","unstructured":"<small>LIU, Y.<\/small>, <small>LI, H.<\/small>, <small>ZHOU, Y.<\/small> and <small>MATSOUAKA, R. A.<\/small> (2024). Average treatment effect on the treated, under lack of positivity. <i>Stat. Methods Med. Res.<\/i> <b>33<\/b> 1689\u20131717.","DOI":"10.1177\/09622802241269646"},{"key":"49","doi-asserted-by":"publisher","unstructured":"<small>MANSKI, C. F.<\/small> (2004). Statistical treatment rules for heterogeneous populations. <i>Econometrica<\/i> <b>72<\/b> 1221\u20131246.","DOI":"10.1111\/j.1468-0262.2004.00530.x"},{"key":"50","unstructured":"<small>MAURER, A.<\/small> and <small>PONTIL, M.<\/small> (2009). Empirical Bernstein bounds and sample variance penalization. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/0907.3740\">0907.3740<\/a>."},{"key":"51","unstructured":"<small>MOU, W.<\/small>, <small>DING, P.<\/small>, <small>WAINWRIGHT, M. J.<\/small> and <small>BARTLETT, P. L.<\/small> (2023). Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2301.06240\">2301.06240<\/a>."},{"key":"54","unstructured":"<small>NACHUM, O.<\/small>, <small>DAI, B.<\/small>, <small>KOSTRIKOV, I.<\/small>, <small>CHOW, Y.<\/small>, <small>LI, L.<\/small> and <small>SCHUURMANS, D.<\/small> (2019). Algaedice: Policy gradient from arbitrary experience. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/1912.02074\">1912.02074<\/a>."},{"key":"55","doi-asserted-by":"crossref","unstructured":"<small>NATARAJAN, B. K.<\/small> (1991). On learning sets and functions. <i>Mach. Learn.<\/i> <b>4<\/b> 67\u201397.","DOI":"10.1007\/BF00114804"},{"key":"56","doi-asserted-by":"crossref","unstructured":"<small>OFFER-WESTORT, M.<\/small>, <small>COPPOCK, A.<\/small> and <small>GREEN, D. P.<\/small> (2021). Adaptive experimental design: Prospects and applications in political science. <i>Amer. J. Polit. Sci.<\/i> <b>65<\/b> 826\u2013844.","DOI":"10.1111\/ajps.12597"},{"key":"57","doi-asserted-by":"publisher","unstructured":"<small>RAKHLIN, A.<\/small>, <small>SRIDHARAN, K.<\/small> and <small>TEWARI, A.<\/small> (2015). Sequential complexities and uniform martingale laws of large numbers. <i>Probab. Theory Related Fields<\/i> <b>161<\/b> 111\u2013153.","DOI":"10.1007\/s00440-013-0545-5"},{"key":"58","unstructured":"<small>RASHIDINEJAD, P.<\/small>, <small>ZHU, B.<\/small>, <small>MA, C.<\/small>, <small>JIAO, J.<\/small> and <small>RUSSELL, S.<\/small> (2022). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 11702\u201311716."},{"key":"59","unstructured":"<small>RASHIDINEJAD, P.<\/small>, <small>ZHU, H.<\/small>, <small>YANG, K.<\/small>, <small>RUSSELL, S.<\/small> and <small>JIAO, J.<\/small> (2022). Optimal conservative offline RL with general function approximation via augmented Lagrangian. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2211.00716\">2211.00716<\/a>."},{"key":"61","doi-asserted-by":"crossref","unstructured":"<small>RUSSO, D.<\/small>, <small>VAN ROY, B.<\/small>, <small>KAZEROUNI, A. OSBAND, I. WEN, Z.<\/small> et al. (2018). A tutorial on Thompson sampling. In <i>Foundations and Trends<\/i>\u00ae <i>in Machine Learning<\/i> <b>11<\/b> 1\u201396.","DOI":"10.1561\/2200000070"},{"key":"63","unstructured":"<small>SCHNABEL, T.<\/small>, <small>SWAMINATHAN, A.<\/small>, <small>SINGH, A.<\/small>, <small>CHANDAK, N.<\/small> and <small>JOACHIMS, T.<\/small> (2016). Recommendations as treatments: Debiasing learning and evaluation. In <i>International Conference on Machine Learning<\/i> 1670\u20131679. PMLR."},{"key":"64","unstructured":"<small>SHI, L.<\/small>, <small>LI, G.<\/small>, <small>WEI, Y.<\/small>, <small>CHEN, Y.<\/small> and <small>CHI, Y.<\/small> (2022). Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2202.13890\">2202.13890<\/a>."},{"key":"65","doi-asserted-by":"crossref","unstructured":"<small>SIMON, R.<\/small> (1977). Adaptive treatment assignment methods and clinical trials. <i>Biometrics<\/i> 743\u2013749.","DOI":"10.2307\/2529473"},{"key":"66","doi-asserted-by":"crossref","unstructured":"<small>STONE, M.<\/small> (1974). Cross-validatory choice and assessment of statistical predictions. <i>J. Roy. Statist. Soc. Ser. B, Methodol.<\/i> <b>36<\/b> 111\u2013147.","DOI":"10.1111\/j.2517-6161.1974.tb00994.x"},{"key":"67","doi-asserted-by":"publisher","unstructured":"<small>STOYE, J.<\/small> (2009). Minimax regret treatment choice with finite samples. <i>J. Econometrics<\/i> <b>151<\/b> 70\u201381.","DOI":"10.1016\/j.jeconom.2009.02.013"},{"key":"68","doi-asserted-by":"publisher","unstructured":"<small>STOYE, J.<\/small> (2012). Minimax regret treatment choice with covariates or with limited validity of experiments. <i>J. Econometrics<\/i> <b>166<\/b> 138\u2013156.","DOI":"10.1016\/j.jeconom.2011.06.012"},{"key":"69","doi-asserted-by":"crossref","unstructured":"<small>SVERDRUP, E.<\/small>, <small>KANODIA, A.<\/small>, <small>ZHOU, Z.<\/small>, <small>ATHEY, S.<\/small> and <small>WAGER, S.<\/small> (2020). policytree: Policy learning via doubly robust empirical welfare maximization over trees. <i>J. Open Sour. Softw.<\/i> <b>5<\/b> 2232.","DOI":"10.21105\/joss.02232"},{"key":"71","unstructured":"<small>SWAMINATHAN, A.<\/small>, <small>KRISHNAMURTHY, A.<\/small>, <small>AGARWAL, A.<\/small>, <small>DUDIK, M.<\/small>, <small>LANGFORD, J.<\/small>, <small>JOSE, D.<\/small> and <small>ZITOUNI, I.<\/small> (2017). Off-policy evaluation for slate recommendation. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>30<\/b>."},{"key":"75","doi-asserted-by":"publisher","unstructured":"<small>VANSCHOREN, J.<\/small>, <small>VAN RIJN, J. N.<\/small>, <small>BISCHL, B.<\/small> and <small>TORGO, L.<\/small> (2013). OpenML: Networked science in machine learning. <i>ACM SIGKDD Explor. Newsl.<\/i> <b>15<\/b> 49\u201360.","DOI":"10.1145\/2641190.2641198"},{"key":"76","unstructured":"<small>VAN DEN BURG, G. J. J.<\/small> and <small>GROENEN, P. J. F.<\/small> (2016). GenSVM: A generalized multiclass support vector machine. <i>J. Mach. Learn. Res.<\/i> <b>17<\/b> Paper No. 225, 42."},{"key":"79","unstructured":"<small>WAUDBY-SMITH, I.<\/small>, <small>WU, L.<\/small>, <small>RAMDAS, A.<\/small>, <small>KARAMPATZIAKIS, N.<\/small> and <small>MINEIRO, P.<\/small> (2022). Anytime-valid off-policy inference for contextual bandits. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2210.10768\">2210.10768<\/a>."},{"key":"80","unstructured":"<small>XIE, T.<\/small>, <small>CHENG, C.-A.<\/small>, <small>JIANG, N.<\/small>, <small>MINEIRO, P.<\/small> and <small>AGARWAL, A.<\/small> (2021). Bellman-consistent pessimism for offline reinforcement learning. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 6683\u20136694."},{"key":"82","unstructured":"<small>XIE, T.<\/small>, <small>MA, Y.<\/small> and <small>WANG, Y.-X.<\/small> (2019). Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>32<\/b>."},{"key":"83","doi-asserted-by":"publisher","unstructured":"<small>XU, Y.<\/small> and <small>ZEEVI, A.<\/small> (2025). Towards optimal problem dependent generalization error bounds in statistical learning theory. <i>Math. Oper. Res.<\/i> <b>50<\/b> 40\u201367.","DOI":"10.1287\/moor.2021.0076"},{"key":"85","doi-asserted-by":"publisher","unstructured":"<small>YANG, S.<\/small> and <small>DING, P.<\/small> (2018). Asymptotic inference of causal effects with observational studies trimmed by the estimated propensity scores. <i>Biometrika<\/i> <b>105<\/b> 487\u2013493.","DOI":"10.1093\/biomet\/asy008"},{"key":"87","unstructured":"<small>ZANETTE, A.<\/small>, <small>WAINWRIGHT, M. J.<\/small> and <small>BRUNSKILL, E.<\/small> (2021). Provable benefits of actor-critic methods for offline reinforcement learning. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 13626\u201313640."},{"key":"88","unstructured":"<small>ZHAN, R.<\/small>, <small>REN, Z.<\/small>, <small>ATHEY, S.<\/small> and <small>ZHOU, Z.<\/small> (2021). Policy learning with adaptively collected data. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2105.02344\">2105.02344<\/a>."},{"key":"89","unstructured":"<small>ZHAN, W.<\/small>, <small>HUANG, B.<\/small>, <small>HUANG, A.<\/small>, <small>JIANG, N.<\/small> and <small>LEE, J.<\/small> (2022). Offline reinforcement learning with realizability and single-policy concentrability. In <i>Conference on Learning Theory<\/i> 2730\u20132775. PMLR."},{"key":"91","unstructured":"<small>ZHANG, R.<\/small>, <small>DAI, B.<\/small>, <small>LI, L.<\/small> and <small>SCHUURMANS, D.<\/small> (2020). Gendice: Generalized offline estimation of stationary values. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2002.09072\">2002.09072<\/a>."},{"key":"92","unstructured":"<small>ZHAO, P.<\/small>, <small>CHAMBAZ, A.<\/small>, <small>JOSSE, J.<\/small> and <small>YANG, S.<\/small> (2023). Positivity-free policy learning with observational data. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2310.06969\">2310.06969<\/a>."},{"key":"94","doi-asserted-by":"publisher","unstructured":"<small>ZHAO, Y. Q.<\/small>, <small>ZENG, D.<\/small>, <small>LABER, E. B.<\/small>, <small>SONG, R.<\/small>, <small>YUAN, M.<\/small> and <small>KOSOROK, M. R.<\/small> (2015). Doubly robust learning for estimating individualized treatment with censored data. <i>Biometrika<\/i> <b>102<\/b> 151\u2013168.","DOI":"10.1093\/biomet\/asu050"},{"key":"45","doi-asserted-by":"crossref","unstructured":"<small>LI, L.<\/small>, <small>CHU, W.<\/small>, <small>LANGFORD, J.<\/small> and <small>SCHAPIRE, R. E.<\/small> (2010). A contextual-bandit approach to personalized news article recommendation. In <i>Proceedings of the<\/i> 19<i>th International Conference on World Wide Web<\/i> 661\u2013670.","DOI":"10.1145\/1772690.1772758"},{"key":"81","unstructured":"<small>XIE, T.<\/small>, <small>JIANG, N.<\/small>, <small>WANG, H.<\/small>, <small>XIONG, C.<\/small> and <small>BAI, Y.<\/small> (2021). Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 27395\u201327407."},{"key":"74","unstructured":"<small>UEHARA, M.<\/small> and <small>SUN, W.<\/small> (2021). Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint. Available at arXiv:<a href=\"https:\/\/arxiv.org\/abs\/2107.06226\">2107.06226<\/a>."},{"key":"84","doi-asserted-by":"crossref","unstructured":"<small>YAN, Y.<\/small>, <small>LI, G.<\/small>, <small>CHEN, Y.<\/small> and <small>FAN, J.<\/small> (2023). The efficacy of pessimism in asynchronous Q-learning. <i>IEEE Trans. Inf. Theory<\/i> <b>69<\/b> 7185\u20137219.","DOI":"10.1109\/TIT.2023.3299840"},{"key":"52","doi-asserted-by":"publisher","unstructured":"<small>MURPHY, S. A.<\/small> (2003). Optimal dynamic treatment regimes. <i>J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> <b>65<\/b> 331\u2013355.","DOI":"10.1111\/1467-9868.00389"},{"key":"35","unstructured":"<small>KALLUS, N.<\/small> (2018). Balanced policy evaluation and learning. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>31<\/b>."},{"key":"86","unstructured":"<small>YIN, M.<\/small> and <small>WANG, Y.-X.<\/small> (2021). Towards instance-optimal offline reinforcement learning with pessimism. <i>Adv. Neural Inf. Process. Syst.<\/i> <b>34<\/b> 4065\u20134078."},{"key":"1","unstructured":"<small>AGRAWAL, S.<\/small> and <small>GOYAL, N.<\/small> (2013). Thompson sampling for contextual bandits with linear payoffs. In <i>International Conference on Machine Learning<\/i> 127\u2013135. PMLR."}],"container-title":["The Annals of Statistics"],"original-title":[],"deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:37:52Z","timestamp":1791391072000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-statistics\/volume-53\/issue-4\/Policy-learning-without-overlap--Pessimism-and-generalized-empirical-Bernsteins\/10.1214\/25-AOS2511.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,1]]},"references-count":97,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,8,1]]}},"URL":"https:\/\/doi.org\/10.1214\/25-aos2511","relation":{},"ISSN":["0090-5364","2168-8966"],"issn-type":[{"value":"0090-5364","type":"print"},{"value":"2168-8966","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,1]]}}}