{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T07:57:11Z","timestamp":1791446231393,"version":"4.3.3"},"reference-count":72,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T00:00:00Z","timestamp":1709769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,11,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve type-I error guarantees, under such optional continuation. We define growth rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a 1-sample safe t-test and the 2\u00d72 contingency table. Sharing Fisherian, Neymanian, and Jeffreys\u2013Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools.<\/jats:p>","DOI":"10.1093\/jrsssb\/qkae011","type":"journal-article","created":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T16:53:24Z","timestamp":1709744004000},"page":"1091-1128","source":"Crossref","is-referenced-by-count":85,"title":["Safe testing"],"prefix":"10.1093","volume":"86","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9832-9936","authenticated-orcid":false,"given":"Peter","family":"Gr\u00fcnwald","sequence":"first","affiliation":[{"name":"Machine Learning Group, Centrum Wiskunde & Informatica , Amsterdam ,","place":["The Netherlands"]},{"name":"Mathematical Institute, Leiden University , Leiden ,","place":["The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rianne","family":"de Heide","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Vrije Universiteit Amsterdam , Amsterdam ,","place":["The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wouter","family":"Koolen","sequence":"additional","affiliation":[{"name":"Machine Learning Group, Centrum Wiskunde & Informatica , Amsterdam ,","place":["The Netherlands"]},{"name":"Statistics Group, University of Twente , Enschede ,","place":["The Netherlands"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,3,7]]},"reference":[{"key":"2024111314115654800_qkae011-B1","author":"Adams","year":"2020"},{"issue":"7748","key":"2024111314115654800_qkae011-B2","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1038\/d41586-019-00857-9","article-title":"Scientists rise up against statistical significance","volume":"567","author":"Amrhein","year":"2019","journal-title":"Nature"},{"key":"2024111314115654800_qkae011-B3","author":"Balsubramani","year":"2016"},{"issue":"6","key":"2024111314115654800_qkae011-B4","doi-asserted-by":"publisher","first-page":"2743","DOI":"10.1109\/18.720554","article-title":"The minimum description length principle in coding and modeling","volume":"44","author":"Barron","year":"1998","journal-title":"IEEE Transactions on Information Theory"},{"issue":"3","key":"2024111314115654800_qkae011-B5","doi-asserted-by":"publisher","first-page":"1550","DOI":"10.1214\/12-AOS1013","article-title":"Criteria for Bayesian model choice with application to variable selection","volume":"40","author":"Bayarri","year":"2012","journal-title":"The Annals of Statistics"},{"key":"2024111314115654800_qkae011-B6","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1038\/s41562-017-0189-z","article-title":"Redefine statistical significance","volume":"2","author":"Benjamin","year":"2018","journal-title":"Nature Human Behaviour"},{"key":"2024111314115654800_qkae011-B7","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4757-4286-2","volume-title":"Statistical decision theory and Bayesian analysis","author":"Berger","year":"1985"},{"issue":"1","key":"2024111314115654800_qkae011-B8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/ss\/1056397485","article-title":"Could Fisher, Jeffreys and Neyman have agreed on testing?","volume":"18","author":"Berger","year":"2003","journal-title":"Statistical Science"},{"issue":"3","key":"2024111314115654800_qkae011-B9","first-page":"307","article-title":"Bayes factors and marginal distributions in invariant situations","volume":"60","author":"Berger","year":"1998","journal-title":"Sankhy\u0101: The Indian Journal of Statistics, Series A"},{"key":"2024111314115654800_qkae011-B10","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-78939-8","volume-title":"Random walk, Brownian motion, and martingales","author":"Bhattacharya","year":"2021"},{"key":"2024111314115654800_qkae011-B11","author":"Breiman","year":"1961"},{"issue":"1","key":"2024111314115654800_qkae011-B12","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/0378-3758(94)90153-8","article-title":"Jeffreys\u2019 prior is asymptotically least favorable under entropy risk","volume":"41","author":"Clarke","year":"1994","journal-title":"Journal of Statistical Planning and Inference"},{"key":"2024111314115654800_qkae011-B13","volume-title":"Elements of information theory","author":"Cover","year":"1991"},{"issue":"1","key":"2024111314115654800_qkae011-B14","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1214\/aop\/1176996454","article-title":"I-divergence geometry of probability distributions and minimization problems","volume":"3","author":"Csisz\u00e1r","year":"1975","journal-title":"Annals of Probability"},{"issue":"6","key":"2024111314115654800_qkae011-B15","doi-asserted-by":"publisher","first-page":"1474","DOI":"10.1109\/TIT.2003.810633","article-title":"Information projections revisited","volume":"49","author":"Csisz\u00e1r","year":"2003","journal-title":"IEEE Transactions on Information Theory"},{"key":"2024111314115654800_qkae011-B16","first-page":"205","article-title":"Information geometry and alternating minimization procedures","volume":"1","author":"Csisz\u00e1r","year":"1984","journal-title":"Statistics and Decisions, Supplemental Issue"},{"key":"2024111314115654800_qkae011-B17","volume-title":"Understanding the new statistics: Effect sizes, confidence and meta-analysis","author":"Cumming","year":"2012"},{"issue":"1","key":"2024111314115654800_qkae011-B18","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1073\/pnas.58.1.66","article-title":"Confidence sequences for mean, variance, and median","volume":"58","author":"Darling","year":"1967","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"2024111314115654800_qkae011-B19","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1111\/sjos.2003.30.issue-1","article-title":"Unified conditional frequentist and Bayesian testing of composite hypotheses","volume":"30","author":"Dass","year":"2003","journal-title":"Scandinavian Journal of Statistics"},{"key":"2024111314115654800_qkae011-B20","author":"Dawid","year":"1997"},{"key":"2024111314115654800_qkae011-B21","doi-asserted-by":"crossref","DOI":"10.1214\/cbms\/1462061029","volume-title":"Group invariance applications in statistics","author":"Eaton","year":"1989"},{"issue":"1\u20134","key":"2024111314115654800_qkae011-B22","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1215\/ijm\/1258059482","article-title":"Robust projections in the class of martingale measures","volume":"50","author":"Fllmer","year":"2006","journal-title":"Illinois Journal of Mathematics"},{"issue":"1\u20133","key":"2024111314115654800_qkae011-B23","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1016\/j.tcs.2005.03.054","article-title":"Uniform test of algorithmic randomness over a general space","volume":"341","author":"G\u00e1cs","year":"2005","journal-title":"Theoretical Computer Science"},{"issue":"4","key":"2024111314115654800_qkae011-B24","doi-asserted-by":"publisher","first-page":"1367","DOI":"10.1214\/009053604000000553","article-title":"Game theory, maximum entropy, minimum discrepancy, and robust Bayesian decision theory","volume":"32","author":"Gr\u00fcnwald","year":"2004","journal-title":"Annals of Statistics"},{"issue":"1","key":"2024111314115654800_qkae011-B25","first-page":"2040","article-title":"Fast rates for general unbounded loss functions: From ERM to generalized Bayes","volume":"21","author":"Gr\u00fcnwald","year":"2020","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2024111314115654800_qkae011-B26","doi-asserted-by":"publisher","first-page":"1930001","DOI":"10.1142\/S2661335219300018","article-title":"Minimum description length revisited","volume":"11","author":"Gr\u00fcnwald","year":"2020","journal-title":"International Journal of Mathematics for Industry"},{"issue":"2","key":"2024111314115654800_qkae011-B27","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1007\/s00780-004-0148-1","article-title":"Robust utility maximization for complete and incomplete market models","volume":"9","author":"Gundel","year":"2005","journal-title":"Finance and Stochastics"},{"issue":"3","key":"2024111314115654800_qkae011-B28","doi-asserted-by":"publisher","first-page":"961","DOI":"10.1214\/20-BA1234","article-title":"Optional stopping with Bayes factors: A categorization and extension of folklore results, with an application to invariant situations","volume":"16","author":"Hendriksen","year":"2021","journal-title":"Bayesian Analysis"},{"issue":"3","key":"2024111314115654800_qkae011-B29","doi-asserted-by":"publisher","first-page":"647","DOI":"10.1093\/biomet\/asab047","article-title":"Valid sequential inference on probability forecast performance","volume":"109","author":"Henzi","year":"2022","journal-title":"Biometrika"},{"issue":"2","key":"2024111314115654800_qkae011-B30","doi-asserted-by":"publisher","first-page":"1055","DOI":"10.1214\/20-AOS1991","article-title":"Uniform, nonparametric, non-asymptotic confidence sequences","volume":"49","author":"Howard","year":"2021","journal-title":"Annals of Statistics"},{"key":"2024111314115654800_qkae011-B31","volume-title":"Theory of probability","author":"Jeffreys","year":"1961","edition":"3rd"},{"issue":"3","key":"2024111314115654800_qkae011-B32","doi-asserted-by":"publisher","first-page":"1806","DOI":"10.1287\/opre.2021.2135","article-title":"Always valid inference: Continuous monitoring of a\/b tests","volume":"70","author":"Johari","year":"2021","journal-title":"Operations Research"},{"issue":"4","key":"2024111314115654800_qkae011-B33","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1002\/bltj.1956.35.issue-4","article-title":"A new interpretation of information rate","volume":"35","author":"Kelly","year":"1956","journal-title":"Bell System Technical Journal"},{"key":"2024111314115654800_qkae011-B34","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.ijar.2021.09.010","article-title":"Log-optimal anytime-valid E-values","volume":"141","author":"Koolen","year":"2021","journal-title":"International Journal of Approximate Reasoning"},{"issue":"2","key":"2024111314115654800_qkae011-B35","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1214\/aos\/1176343406","article-title":"On confidence sequences","volume":"4","author":"Lai","year":"1976","journal-title":"The Annals of Statistics"},{"issue":"1","key":"2024111314115654800_qkae011-B36","article-title":"Martingales in sequential analysis and time series, 1945\u20131985","volume":"5","author":"Lai","year":"2009","journal-title":"Electronic Journal for History of Probability and Statistics"},{"key":"2024111314115654800_qkae011-B37","volume-title":"Testing statistical hypotheses","author":"Lehmann","year":"2005"},{"issue":"1","key":"2024111314115654800_qkae011-B38","first-page":"337","article-title":"Uniform tests of randomness","volume":"227","author":"Levin","year":"1976","journal-title":"Soviet Mathematics Doklady"},{"key":"2024111314115654800_qkae011-B39","author":"Li","year":"1999"},{"issue":"481","key":"2024111314115654800_qkae011-B40","doi-asserted-by":"publisher","first-page":"410","DOI":"10.1198\/016214507000001337","article-title":"Mixtures of g-priors for Bayesian variable selection","volume":"103","author":"Liang","year":"2008","journal-title":"Journal of the American Statistical Association"},{"issue":"Suppl. 1","key":"2024111314115654800_qkae011-B41","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1080\/00031305.2018.1527253","article-title":"Abandon statistical significance","volume":"73","author":"McShane","year":"2019","journal-title":"The American Statistician"},{"issue":"1","key":"2024111314115654800_qkae011-B42","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1109\/TIT.2023.3330187","article-title":"Tight concentrations and confidence sequences from the regret of universal portfolio","volume":"70","author":"Orabona","year":"2024","journal-title":"IEEE Transactions on Information Theory"},{"issue":"3","key":"2024111314115654800_qkae011-B43","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1111\/insr.12355","article-title":"Likelihood, replicability and Robbins\u2019 confidence sequences","volume":"88","author":"Pace","year":"2019","journal-title":"International Statistical Review"},{"key":"2024111314115654800_qkae011-B44","author":"P\u00e9rez-Ortiz","year":"2022"},{"issue":"4","key":"2024111314115654800_qkae011-B45","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1109\/TIT.1975.1055416","article-title":"Random coding strategies for minimum entropy","volume":"21","author":"Posner","year":"1975","journal-title":"IEEE Transactions on Information Theory"},{"key":"2024111314115654800_qkae011-B46","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1016\/j.ijar.2021.06.17","article-title":"Testing exchangeability: Fork-convexity, supermartingales and e-processes","volume":"141","author":"Ramdas","year":"2021","journal-title":"International Journal of Approximate Reasoning"},{"issue":"5","key":"2024111314115654800_qkae011-B47","doi-asserted-by":"publisher","first-page":"1397","DOI":"10.1214\/aoms\/1177696786","article-title":"Statistical methods related to the law of the iterated logarithm","volume":"41","author":"Robbins","year":"1970","journal-title":"The Annals of Mathematical Statistics"},{"issue":"2","key":"2024111314115654800_qkae011-B48","doi-asserted-by":"publisher","first-page":"225","DOI":"10.3758\/PBR.16.2.225","article-title":"Bayesian t-tests for accepting and rejecting the null hypothesis","volume":"16","author":"Rouder","year":"2009","journal-title":"Psychonomic Bulletin & Review"},{"key":"2024111314115654800_qkae011-B49","volume-title":"Statistical evidence: A likelihood paradigm","author":"Royall","year":"1997"},{"issue":"45","key":"2024111314115654800_qkae011-B50","doi-asserted-by":"publisher","first-page":"760","DOI":"10.2307\/2669456","article-title":"On the probability of observing misleading statistical evidence","volume":"95","author":"Royall","year":"2000","journal-title":"Journal of the American Statistical Association"},{"key":"2024111314115654800_qkae011-B51","first-page":"263","article-title":"On the minimum discrimination information theorem","volume":"1","author":"R\u00fcschendorf","year":"1984","journal-title":"Statistics and Decisions, Supplemental Issue"},{"issue":"1","key":"2024111314115654800_qkae011-B52","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1198\/000313001300339950","article-title":"Calibration of p-values for testing precise null hypotheses","volume":"55","author":"Sellke","year":"2001","journal-title":"The American Statistician"},{"issue":"2","key":"2024111314115654800_qkae011-B53","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1111\/rssa.12647","article-title":"Testing by betting: A strategy for statistical and scientific communication","volume":"184","author":"Shafer","year":"2021","journal-title":"Journal of the Royal Statistical Society, Series A"},{"issue":"1","key":"2024111314115654800_qkae011-B54","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1214\/10-STS347","article-title":"Test martingales, Bayes factors and p-values","volume":"26","author":"Shafer","year":"2011","journal-title":"Statistical Science"},{"key":"2024111314115654800_qkae011-B55","volume-title":"Game-theoretic probability: Theory and applications to prediction, science and finance","author":"Shafer","year":"2019"},{"key":"2024111314115654800_qkae011-B56","volume-title":"Sequential analysis: Tests and confidence intervals","author":"Siegmund","year":"2013"},{"issue":"549","key":"2024111314115654800_qkae011-B57","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research","article-title":"ALL-IN meta-analysis: Breathing life into living systematic reviews version 1; peer review: 1 approved 2 approved with reservations","volume":"11","author":"Ter Schure","year":"2022","journal-title":"F1000Research"},{"key":"2024111314115654800_qkae011-B58","author":"Ter Schure","year":"2022"},{"key":"2024111314115654800_qkae011-B59","author":"Ter Schure","year":"2023"},{"issue":"1","key":"2024111314115654800_qkae011-B60","first-page":"8","article-title":"Information-theoretical optimization techniques","volume":"15","author":"Tops\u00f8e","year":"1979","journal-title":"Kybernetika"},{"key":"2024111314115654800_qkae011-B61","doi-asserted-by":"publisher","DOI":"10.1016\/j.jspi.2023.106116","article-title":"Generic e-variables for exact sequential k-sample tests that allow for optional stopping","volume":"230","author":"Turner","year":"2024","journal-title":"Statistical Planning and Inference"},{"key":"2024111314115654800_qkae011-B62","author":"Turner","year":"2022"},{"issue":"3","key":"2024111314115654800_qkae011-B63","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1111\/j.1467-9868.2011.01025.x","article-title":"Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC\u2013BIC dilemma","volume":"74","author":"Van Erven","year":"2012","journal-title":"Journal of the Royal Statistical Society Series B: Statistical Methodology"},{"key":"2024111314115654800_qkae011-B64","volume-title":"Monographies des Probabilit\u00e9s","author":"Ville","year":"1939"},{"issue":"2","key":"2024111314115654800_qkae011-B65","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1111\/j.2517-6161.1993.tb01904.x","article-title":"A logic of probability, with application to the foundations of statistics","volume":"55","author":"Vovk","year":"1993","journal-title":"Journal of the Royal Statistical Society, Series B"},{"issue":"3","key":"2024111314115654800_qkae011-B66","doi-asserted-by":"publisher","first-page":"1736","DOI":"10.1214\/20-AOS2020","article-title":"E-values: Calibration, combination, and applications","volume":"49","author":"Vovk","year":"2021","journal-title":"Annals of Statistics"},{"key":"2024111314115654800_qkae011-B67","volume-title":"Sequential analysis","author":"Wald","year":"1947"},{"issue":"3","key":"2024111314115654800_qkae011-B68","doi-asserted-by":"publisher","first-page":"822","DOI":"10.1111\/rssb.12489","article-title":"False discovery rate control with e-values","volume":"84","author":"Wang","year":"2022","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"2","key":"2024111314115654800_qkae011-B69","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1080\/00031305.2016.1154108","article-title":"The ASA statement on p-values: Context, process, and purpose","volume":"70","author":"Wasserstein","year":"2016","journal-title":"The American Statistician"},{"issue":"1","key":"2024111314115654800_qkae011-B70","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/jrsssb\/qkad009","article-title":"Estimating means of bounded random variables by betting","volume":"86","author":"Waudby-Smith","year":"2024","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"key":"2024111314115654800_qkae011-B71","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511813658","volume-title":"Probability with martingales","author":"Williams","year":"1991"},{"issue":"6","key":"2024111314115654800_qkae011-B72","doi-asserted-by":"publisher","first-page":"062118","DOI":"10.1103\/PhysRevA.84.062118","article-title":"Asymptotically optimal data analysis for rejecting local realism","volume":"84","author":"Zhang","year":"2011","journal-title":"Physical Review A"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jrsssb\/advance-article-pdf\/doi\/10.1093\/jrsssb\/qkae011\/59646264\/qkae011.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/5\/1091\/60648648\/qkae011.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/86\/5\/1091\/60648648\/qkae011.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,13]],"date-time":"2024-11-13T14:12:28Z","timestamp":1731507148000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/86\/5\/1091\/7623686"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,7]]},"references-count":72,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,3,7]]},"published-print":{"date-parts":[[2024,11,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jrsssb\/qkae011","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,11]]},"published":{"date-parts":[[2024,3,7]]}}}