{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T19:02:23Z","timestamp":1781204543123,"version":"3.54.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011393","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,9,11]],"date-time":"2023-09-11T00:00:00Z","timestamp":1694390400000}}],"reference-count":38,"publisher":"Public Library of Science (PLoS)","issue":"8","license":[{"start":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T00:00:00Z","timestamp":1693267200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Institute for Health and Care Research (NIHR) Health Protection Research Unit","award":["NIHR200908"],"award-info":[{"award-number":["NIHR200908"]}]},{"name":"National Institute for Health and Care Research (NIHR) Health Protection Research Unit","award":["NIHR200908"],"award-info":[{"award-number":["NIHR200908"]}]},{"DOI":"10.13039\/501100001656","name":"Helmholtz-Gemeinschaft","doi-asserted-by":"publisher","award":["512483310"],"award-info":[{"award-number":["512483310"]}],"id":[{"id":"10.13039\/501100001656","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["210758\/Z\/18\/Z"],"award-info":[{"award-number":["210758\/Z\/18\/Z"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"name":"MRC Centre for Global Infectious Disease Analysis","award":["MR\/R015600\/1"],"award-info":[{"award-number":["MR\/R015600\/1"]}]},{"DOI":"10.13039\/501100000691","name":"Academy of Medical Sciences","doi-asserted-by":"publisher","award":["SBF005\\1044"],"award-info":[{"award-number":["SBF005\\1044"]}],"id":[{"id":"10.13039\/501100000691","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Institute for Health and Care Research (NIHR) Health Protection Research Unit","award":["NIHR200908"],"award-info":[{"award-number":["NIHR200908"]}]},{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["101003688"],"award-info":[{"award-number":["101003688"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["210758\/Z\/18\/Z"],"award-info":[{"award-number":["210758\/Z\/18\/Z"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Institute for Health and Care Research (NIHR) Health Protection Research Unit","award":["NIHR200908"],"award-info":[{"award-number":["NIHR200908"]}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011393","type":"journal-article","created":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T17:54:36Z","timestamp":1693331676000},"page":"e1011393","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":52,"title":["Scoring epidemiological forecasts on transformed scales"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7750-5280","authenticated-orcid":true,"given":"Nikos I.","family":"Bosse","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sam","family":"Abbott","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anne","family":"Cori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2383-5305","authenticated-orcid":true,"given":"Edwin","family":"van Leeuwen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johannes","family":"Bracher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2842-3406","authenticated-orcid":true,"given":"Sebastian","family":"Funk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2023,8,29]]},"reference":[{"issue":"22","key":"pcbi.1011393.ref001","doi-asserted-by":"crossref","first-page":"3443","DOI":"10.1002\/sim.7363","article-title":"Probabilistic Forecasting in Infectious Disease Epidemiology: The 13th Armitage Lecture","volume":"36","author":"L Held","year":"2017","journal-title":"Statistics in Medicine"},{"issue":"6","key":"pcbi.1011393.ref002","doi-asserted-by":"crossref","first-page":"839","DOI":"10.2105\/AJPH.2022.306831","article-title":"Collaborative Hubs: Making the Most of Predictive Epidemic Modeling","volume":"112","author":"NG Reich","year":"2022","journal-title":"American Journal of Public Health"},{"issue":"1","key":"pcbi.1011393.ref003","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1146\/annurev-financial-110217-022713","article-title":"Forecasting Methods in Finance","volume":"10","author":"A Timmermann","year":"2018","journal-title":"Annual Review of Financial Economics"},{"issue":"5746","key":"pcbi.1011393.ref004","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1126\/science.1115255","article-title":"Weather Forecasting with Ensemble Methods","volume":"310","author":"T Gneiting","year":"2005","journal-title":"Science"},{"issue":"477","key":"pcbi.1011393.ref005","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly Proper Scoring Rules, Prediction, and Estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"Journal of the American Statistical Association"},{"issue":"1","key":"pcbi.1011393.ref006","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1111\/j.2517-6161.1952.tb00104.x","article-title":"Rational Decisions","volume":"14","author":"IJ Good","year":"1952","journal-title":"Journal of the Royal Statistical Society Series B (Methodological)"},{"issue":"1","key":"pcbi.1011393.ref007","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02562681","article-title":"Scoring Rules and the Evaluation of Probabilities","volume":"5","author":"RL Winkler","year":"1996","journal-title":"Test"},{"issue":"2","key":"pcbi.1011393.ref008","doi-asserted-by":"crossref","first-page":"e1008618","DOI":"10.1371\/journal.pcbi.1008618","article-title":"Evaluating Epidemic Forecasts in an Interval Format","volume":"17","author":"J Bracher","year":"2021","journal-title":"PLoS computational biology"},{"issue":"48","key":"pcbi.1011393.ref009","doi-asserted-by":"crossref","first-page":"24268","DOI":"10.1073\/pnas.1909865116","article-title":"An Open Challenge to Advance Probabilistic Forecasting for Dengue Epidemics","volume":"116","author":"MA Johansson","year":"2019","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1011393.ref010","unstructured":"Cramer E, Reich NG, Wang SY, Niemi J, Hannan A, House K, et al. COVID-19 Forecast Hub: 4 December 2020 Snapshot; 2020."},{"key":"pcbi.1011393.ref011","article-title":"Evaluation of Individual and Ensemble Probabilistic Forecasts of COVID-19 Mortality in the US","author":"E Cramer","year":"2021","journal-title":"medRxiv"},{"key":"pcbi.1011393.ref012","article-title":"Predictive Performance of Multi-Model Ensemble Forecasts of COVID-19 across European Nation","author":"K Sherratt","year":"2022","journal-title":"Europe PMC"},{"key":"pcbi.1011393.ref013","article-title":"Short-Term Forecasting of COVID-19 in Germany and Poland during the Second Wave\u2014a Preregistered Study","author":"J Bracher","year":"2021","journal-title":"medRxiv"},{"key":"pcbi.1011393.ref014","article-title":"National and subnational short-term forecasting of COVID-19 in Germany and Poland, early 2021","author":"J Bracher","year":"2022","journal-title":"Communications Medicine"},{"key":"pcbi.1011393.ref015","unstructured":"CDC. Cdcepi\/Flusight-forecast-data; 2022. CDC Epidemic Prediction Initiative."},{"key":"pcbi.1011393.ref016","article-title":"Practical Considerations for Measuring the Effective Reproductive Number, Rt","author":"KM Gostic","year":"2020","journal-title":"medRxiv"},{"issue":"1947","key":"pcbi.1011393.ref017","doi-asserted-by":"crossref","first-page":"20201556","DOI":"10.1098\/rspb.2020.1556","article-title":"Speed and Strength of an Epidemic Intervention","volume":"288","author":"J Dushoff","year":"2021","journal-title":"Proceedings of the Royal Society B: Biological Sciences"},{"issue":"1","key":"pcbi.1011393.ref018","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1214\/22-STS864","article-title":"Local Scale Invariance and Robustness of Proper Scoring Rules","volume":"38","author":"D Bolin","year":"2023","journal-title":"Statistical Science"},{"issue":"2","key":"pcbi.1011393.ref019","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1002\/(SICI)1099-131X(199903)18:2<111::AID-FOR713>3.0.CO;2-C","article-title":"Evaluating Volatility and Interval Forecasts","volume":"18","author":"JW Taylor","year":"1999","journal-title":"Journal of Forecasting"},{"key":"pcbi.1011393.ref020","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/j.econlet.2014.11.008","article-title":"Log versus Level in VAR Forecasting: 42 Million Empirical Answers\u2014Expect the Unexpected","volume":"126","author":"J Mayr","year":"2015","journal-title":"Economics Letters"},{"issue":"3","key":"pcbi.1011393.ref021","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1007\/s00477-013-0768-0","article-title":"Stochastic Rainfall-Runoff Forecasting: Parameter Estimation, Multi-Step Prediction, and Evaluation of Overflow Risk","volume":"28","author":"R L\u00f6we","year":"2014","journal-title":"Stochastic Environmental Research and Risk Assessment"},{"key":"pcbi.1011393.ref022","first-page":"505","volume":"14","author":"GA Fuglstad","year":"2015","journal-title":"Does Non-Stationary Spatial Data Always Require Non-Stationary Random Fields? Spatial Statistics"},{"issue":"1","key":"pcbi.1011393.ref023","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1214\/aoms\/1177729884","article-title":"Some Principles of the Theory of Testing Hypotheses","volume":"21","author":"EL Lehmann","year":"1950","journal-title":"The Annals of Mathematical Statistics"},{"issue":"2","key":"pcbi.1011393.ref024","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.jeconom.2011.04.001","article-title":"Likelihood-Based Scoring Rules for Comparing Density Forecasts in Tails","volume":"163","author":"C Diks","year":"2011","journal-title":"Journal of Econometrics"},{"key":"pcbi.1011393.ref025","unstructured":"European Covid-19 Forecast Hub. European Covid-19 Forecast Hub; 2021. https:\/\/covid19forecasthub.eu\/."},{"issue":"494","key":"pcbi.1011393.ref026","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1198\/jasa.2011.r10138","article-title":"Making and Evaluating Point Forecasts","volume":"106","author":"T Gneiting","year":"2011","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"pcbi.1011393.ref027","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/0305-0483(86)90013-7","article-title":"A pragmatic view of accuracy measurement in forecasting","volume":"14","author":"BE Flores","year":"1986","journal-title":"Omega"},{"issue":"1609","key":"pcbi.1011393.ref028","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1098\/rspb.2006.3754","article-title":"How Generation Intervals Shape the Relationship between Growth Rates and Reproductive Numbers","volume":"274","author":"J Wallinga","year":"2007","journal-title":"Proceedings of the Royal Society B: Biological Sciences"},{"issue":"1","key":"pcbi.1011393.ref029","doi-asserted-by":"crossref","first-page":"68","DOI":"10.2307\/2983678","article-title":"The Square Root Transformation in Analysis of Variance","volume":"3","author":"MS Bartlett","year":"1936","journal-title":"Supplement to the Journal of the Royal Statistical Society"},{"key":"pcbi.1011393.ref030","volume-title":"Generalized Linear Models With Examples in R","author":"PK Dunn","year":"2018"},{"key":"pcbi.1011393.ref031","unstructured":"Bell\u00e9go C, Benatia D, Pape L. Dealing with Logs and Zeros in Regression Models; 2022."},{"key":"pcbi.1011393.ref032","unstructured":"Lerch S, Thorarinsdottir TL, Ravazzolo F, Gneiting T. Forecaster\u2019s Dilemma: Extreme Events and Forecast Evaluation; 2015."},{"key":"pcbi.1011393.ref033","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing; 2022. Available from: https:\/\/www.R-project.org\/."},{"key":"pcbi.1011393.ref034","article-title":"Evaluating Forecasts with Scoringutils in R","author":"NI Bosse","year":"2022","journal-title":"arXiv"},{"key":"pcbi.1011393.ref035","doi-asserted-by":"crossref","unstructured":"Abbott S, Sherratt K, Bosse N, Gruson H, Bracher J, Funk S. Evaluating an Epidemiologically Motivated Surrogate Model of a Multi-Model Ensemble; 2022.","DOI":"10.1101\/2022.10.12.22280917"},{"issue":"2","key":"pcbi.1011393.ref036","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1111\/j.2517-6161.1964.tb00553.x","article-title":"An Analysis of Transformations","volume":"26","author":"GEP Box","year":"1964","journal-title":"Journal of the Royal Statistical Society Series B (Methodological)"},{"key":"pcbi.1011393.ref037","doi-asserted-by":"crossref","unstructured":"Srivastava A, Singh S, Lee F. Shape-Based Evaluation of Epidemic Forecasts; 2022.","DOI":"10.1109\/BigData55660.2022.10020895"},{"key":"pcbi.1011393.ref038","doi-asserted-by":"crossref","unstructured":"Abbott S, Hellewell J, Sherratt K, Gostic K, Hickson J, Badr HS, et al. EpiNow2: Estimate Real-Time Case Counts and Time-Varying Epidemiological Parameters; 2020.","DOI":"10.32614\/CRAN.package.EpiNow2"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011393","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,9,11]],"date-time":"2023-09-11T00:00:00Z","timestamp":1694390400000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011393","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,27]],"date-time":"2024-10-27T01:54:26Z","timestamp":1729994066000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011393"}},"subtitle":[],"editor":[{"given":"James M","family":"McCaw","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2023,8,29]]},"references-count":38,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,8,29]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011393","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1011393","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,29]]}}}