{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T07:28:11Z","timestamp":1791876491265,"version":"4.3.4"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2021,9,27]],"date-time":"2021-09-27T00:00:00Z","timestamp":1632700800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,8,24]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts numerical scores such that a correct forecast achieves a minimal expected score. In this paper, we construct e-values for testing the statistical significance of score differences of competing forecasts in sequential settings. E-values have been proposed as an alternative to $p$-values for hypothesis testing, and they can easily be transformed into conservative $p$-values by taking the multiplicative inverse. The e-values proposed in this article are valid in finite samples without any assumptions on the data-generating processes. They also allow optional stopping, so a forecast user may decide to interrupt evaluation, taking into account the available data at any time, and still draw statistically valid inference, which is generally not true for classical $p$-value-based tests. In a case study on post-processing of precipitation forecasts, state-of-the-art forecast dominance tests and e-values lead to the same conclusions.<\/jats:p>","DOI":"10.1093\/biomet\/asab047","type":"journal-article","created":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T11:15:28Z","timestamp":1632136528000},"page":"647-663","source":"Crossref","is-referenced-by-count":33,"title":["Valid sequential inference on probability forecast performance"],"prefix":"10.1093","volume":"109","author":[{"given":"Alexander","family":"Henzi","sequence":"first","affiliation":[{"name":"Institute of Mathematical Statistics and Actuarial Science, University of Bern , Alpeneggstrasse 22, 3012 Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5916-9746","authenticated-orcid":false,"given":"Johanna F","family":"Ziegel","sequence":"additional","affiliation":[{"name":"Institute of Mathematical Statistics and Actuarial Science, University of Bern , Alpeneggstrasse 22, 3012 Bern, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,9,27]]},"reference":[{"key":"2022082420280984000_B1","doi-asserted-by":"crossref","first-page":"1076","DOI":"10.1175\/MWR2905.1","article-title":"A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems","volume":"133","author":"Buizza,","year":"2005","journal-title":"Mon. Weather Rev."},{"key":"2022082420280984000_B2","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1080\/07350015.1995.10524599","article-title":"Comparing predictive accuracy","volume":"13","author":"Diebold,","year":"1995","journal-title":"J. Bus. Econ. Statist."},{"key":"2022082420280984000_B3","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1111\/rssb.12154","article-title":"Of quantiles and expectiles: Consistent scoring functions, Choquet representations and forecast rankings","volume":"78","author":"Ehm,","year":"2016","journal-title":"J. R. Statist. Soc. B."},{"key":"2022082420280984000_B4","doi-asserted-by":"crossref","first-page":"3758","DOI":"10.1214\/18-EJS1495","article-title":"Forecast dominance testing via sign randomization","volume":"12","author":"Ehm,","year":"2018","journal-title":"Electron. J. Statist."},{"key":"2022082420280984000_B5","doi-asserted-by":"crossref","first-page":"1545","DOI":"10.1111\/j.1468-0262.2006.00718.x","article-title":"Tests of conditional predictive ability","volume":"74","author":"Giacomini,","year":"2006","journal-title":"Econometrica"},{"key":"2022082420280984000_B6","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1198\/jasa.2011.r10138","article-title":"Making and evaluating point forecasts","volume":"106","author":"Gneiting,","year":"2011","journal-title":"J. Am. Statist. Assoc."},{"key":"2022082420280984000_B7","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/j.1467-9868.2007.00587.x","article-title":"Probabilistic forecasts, calibration and sharpness","volume":"69","author":"Gneiting,","year":"2007","journal-title":"J. R. Statist. Soc. B"},{"key":"2022082420280984000_B8","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"Gneiting,","year":"2007","journal-title":"J. Am. Statist. Assoc."},{"key":"2022082420280984000_B9","doi-asserted-by":"crossref","first-page":"1747","DOI":"10.1214\/13-EJS823","article-title":"Combining predictive distributions","volume":"7","author":"Gneiting,","year":"2013","journal-title":"Electron. J. Statist."},{"key":"2022082420280984000_B10","doi-asserted-by":"crossref","DOI":"10.1109\/ITA50056.2020.9244948","article-title":"Safe testing","volume-title":"2020 Information Theory and Applications Workshop (ITA)","author":"Gr\u00fcnwald,","year":"2020"},{"key":"2022082420280984000_B11","doi-asserted-by":"crossref","first-page":"963","DOI":"10.1111\/rssb.12450","article-title":"Isotonic distributional regression","volume":"83","author":"Henzi,","year":"2021","journal-title":"J. R. Statist. Soc. B"},{"key":"2022082420280984000_B12","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1002\/j.1538-7305.1956.tb03809.x","article-title":"A new interpretation of information rate","volume":"35","author":"Kelly Jr,","year":"1956","journal-title":"Bell System Tech. J."},{"key":"2022082420280984000_B13","doi-asserted-by":"crossref","first-page":"2356","DOI":"10.1214\/11-AOS902","article-title":"Evaluating probability forecasts","volume":"39","author":"Lai,","year":"2011","journal-title":"Ann. Statist."},{"key":"2022082420280984000_B14","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1080\/07350015.2018.1506926","article-title":"HAR inference: Recommendations for practice","volume":"36","author":"Lazarus,","year":"2018","journal-title":"J. Bus. Econ. Statist."},{"key":"2022082420280984000_B15","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1214\/16-STS588","article-title":"Forecaster\u2019s dilemma: Extreme events and forecast evaluation","volume":"32","author":"Lerch,","year":"2017","journal-title":"Statist. Sci."},{"key":"2022082420280984000_B16","doi-asserted-by":"crossref","first-page":"3515","DOI":"10.1016\/j.jcp.2007.02.014","article-title":"Ensemble forecasting","volume":"227","author":"Leutbecher,","year":"2008","journal-title":"J. Comp. Phys."},{"key":"2022082420280984000_B17","doi-asserted-by":"crossref","first-page":"3003","DOI":"10.1175\/MWR-D-13-00355.1","article-title":"Extending extended logistic regression: Extended versus separate versus ordered versus censored","volume":"142","author":"Messner,","year":"2014","journal-title":"Mon. Weather Rev."},{"key":"2022082420280984000_B18","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1002\/qj.49712252905","article-title":"The ECMWF ensemble prediction system: Methodology and validation","volume":"122","author":"Molteni,","year":"1996","journal-title":"Q. J. R. Meteorol. Soc."},{"key":"2022082420280984000_B19","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1080\/07350015.2019.1585256","article-title":"Comparing possibly misspecified forecasts","volume":"38","author":"Patton,","year":"2020","journal-title":"J. Bus. Econ. Statist."},{"key":"2022082420280984000_B20","article-title":"Admissible anytime-valid sequential inference must rely on nonnegative martingales","author":"Ramdas,","year":"2020"},{"key":"2022082420280984000_B21","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1111\/j.1467-9868.2009.00726.x","article-title":"Combining probability forecasts","volume":"72","author":"Ranjan,","year":"2010","journal-title":"J. R. Statist. Soc. B"},{"key":"2022082420280984000_B22","doi-asserted-by":"crossref","first-page":"1856","DOI":"10.1214\/aos\/1176347398","article-title":"A general method for comparing probability assessors","volume":"17","author":"Schervish,","year":"1989","journal-title":"Ann. Statist."},{"key":"2022082420280984000_B23","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1080\/01621459.1993.10594328","article-title":"On testing the validity of sequential probability forecasts","volume":"88","author":"Seillier-Moiseiwitsch,","year":"1993","journal-title":"J. Am. Statist. Assoc."},{"key":"2022082420280984000_B24","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1111\/rssa.12647","article-title":"Testing by betting: A strategy for statistical and scientific communication","volume":"184","author":"Shafer,","year":"2021","journal-title":"J. R. Statist. Soc. A"},{"key":"2022082420280984000_B25","volume-title":"Statistical Postprocessing of Ensemble Forecasts","author":"Vannitsem,","year":"2018"},{"key":"2022082420280984000_B26","article-title":"True and false discoveries with independent e-values","author":"Vovk,","year":"2020"},{"key":"2022082420280984000_B27","doi-asserted-by":"crossref","first-page":"1739","DOI":"10.1214\/20-AOS2020","article-title":"E-values: Calibration, combination, and applications","volume":"49","author":"Vovk,","year":"2021","journal-title":"Ann. Statist."},{"key":"2022082420280984000_B28","article-title":"False discovery rate control with e-values","author":"Wang,","year":"2020"},{"key":"2022082420280984000_B29","article-title":"Estimating means of bounded random variables by betting","author":"Waudby-Smith,","year":"2021"},{"key":"2022082420280984000_B30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02562681","article-title":"Scoring rules and the evaluation of probabilities","volume":"5","author":"Winkler,","year":"1996","journal-title":"Test"},{"key":"2022082420280984000_B31","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1016\/j.ijforecast.2020.09.004","article-title":"Testing forecast accuracy of expectiles and quantiles with the extremal consistent loss functions","volume":"37","author":"Yen,","year":"2021","journal-title":"Int. J. Forecast."},{"key":"2022082420280984000_B32","article-title":"Can two forecasts have the same conditional expected accuracy","author":"Zhu,","year":"2020"}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/biomet\/advance-article-pdf\/doi\/10.1093\/biomet\/asab047\/42248623\/asab047.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/109\/3\/647\/45512191\/asab047.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/109\/3\/647\/45512191\/asab047.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,8]],"date-time":"2024-09-08T15:48:34Z","timestamp":1725810514000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/109\/3\/647\/6375942"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,27]]},"references-count":32,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,9,27]]},"published-print":{"date-parts":[[2022,8,24]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asab047","relation":{},"ISSN":["0006-3444","1464-3510"],"issn-type":[{"value":"0006-3444","type":"print"},{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,9,1]]},"published":{"date-parts":[[2021,9,27]]}}}