{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T16:02:03Z","timestamp":1781625723377,"version":"3.54.5"},"reference-count":38,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2021,2,12]],"date-time":"2021-02-12T00:00:00Z","timestamp":1613088000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>For practical reasons, many forecasts of case, hospitalization, and death counts in the context of the current Coronavirus Disease 2019 (COVID-19) pandemic are issued in the form of central predictive intervals at various levels. This is also the case for the forecasts collected in the<jats:italic>COVID-19 Forecast Hub<\/jats:italic>(<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/covid19forecasthub.org\/\" xlink:type=\"simple\">https:\/\/covid19forecasthub.org\/<\/jats:ext-link>). Forecast evaluation metrics like the logarithmic score, which has been applied in several infectious disease forecasting challenges, are then not available as they require full predictive distributions. This article provides an overview of how established methods for the evaluation of quantile and interval forecasts can be applied to epidemic forecasts in this format. Specifically, we discuss the computation and interpretation of the weighted interval score, which is a proper score that approximates the continuous ranked probability score. It can be interpreted as a generalization of the absolute error to probabilistic forecasts and allows for a decomposition into a measure of sharpness and penalties for over- and underprediction.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1008618","type":"journal-article","created":{"date-parts":[[2021,2,12]],"date-time":"2021-02-12T21:13:00Z","timestamp":1613164380000},"page":"e1008618","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":277,"title":["Evaluating epidemic forecasts in an interval format"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3777-1410","authenticated-orcid":true,"given":"Johannes","family":"Bracher","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Evan L.","family":"Ray","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9397-3271","authenticated-orcid":true,"given":"Tilmann","family":"Gneiting","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3503-9899","authenticated-orcid":true,"given":"Nicholas G.","family":"Reich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2021,2,12]]},"reference":[{"issue":"683","key":"pcbi.1008618.ref001","article-title":"Collaborative efforts to forecast seasonal influenza in the United States, 2015\u20132016","volume":"9","author":"CJ McGowan","year":"2019","journal-title":"Sci Rep"},{"issue":"48","key":"pcbi.1008618.ref002","doi-asserted-by":"crossref","first-page":"24268","DOI":"10.1073\/pnas.1909865116","article-title":"An open challenge to advance probabilistic forecasting for dengue epidemics","volume":"116","author":"MA Johansson","year":"2019","journal-title":"Proc Natl Acad Sci"},{"issue":"477","key":"pcbi.1008618.ref003","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"J Am Stat Assoc"},{"key":"pcbi.1008618.ref004","unstructured":"UMass-Amherst Influenza Forecasting Center of Excellence. COVID-19 Forecast Hub; 2020. Accessible online at https:\/\/github.com\/reichlab\/covid19-forecast-hub. Last accessed 9 October 2020."},{"key":"pcbi.1008618.ref005","article-title":"COVID-19 Forecast Hub Consortium","author":"EL Ray","year":"2020","journal-title":"Ensemble forecasts of coronavirus disease 2019 (COVID-19) in the U.S. Preprint, medRxiv"},{"key":"pcbi.1008618.ref006","unstructured":"Centers for Disease Control and Prevention. COVID-19 Forecasting: Background Information. Accessible online at https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/cases-updates\/forecasting.html. Last accessed 9 October 2020."},{"key":"pcbi.1008618.ref007","unstructured":"Centers for Disease Control and Prevention. Forecast the 2019\u20132020 Influenza Season Collaborative Challenge; 2019. Accessible online at https:\/\/predict.cdc.gov\/api\/v1\/attachments\/flusight_2019\u22122020\/2019\u22122020_flusight_national_regional_guidance_final.docx."},{"key":"pcbi.1008618.ref008","unstructured":"Centers for Disease Control and Prevention. Forecast the 2018\u20132019 Influenza Season Collaborative Challenge; 2018. Accessible online at https:\/\/predict.cdc.gov\/api\/v1\/attachments\/flusight%202018%E2%80%932019\/flu_challenge_2018-19_tentativefinal_9.18.18.docx"},{"issue":"8","key":"pcbi.1008618.ref009","doi-asserted-by":"crossref","first-page":"3146","DOI":"10.1073\/pnas.1812594116","article-title":"A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States","volume":"116","author":"NG Reich","year":"2019","journal-title":"Proc Natl Acad Sci"},{"issue":"42","key":"pcbi.1008618.ref010","doi-asserted-by":"crossref","first-page":"20809","DOI":"10.1073\/pnas.1912147116","article-title":"On the multibin logarithmic score used in the FluSight competitions","volume":"116","author":"J. Bracher","year":"2019","journal-title":"Proc Natl Acad Sci"},{"issue":"42","key":"pcbi.1008618.ref011","doi-asserted-by":"crossref","first-page":"20811","DOI":"10.1073\/pnas.1912694116","article-title":"Reply to Bracher: Scoring probabilistic forecasts to maximize public health interpretability","volume":"116","author":"NG Reich","year":"2019","journal-title":"Proc Natl Acad Sci"},{"issue":"2","key":"pcbi.1008618.ref012","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/j.1467-9868.2007.00587.x","article-title":"Probabilistic forecasts, calibration and sharpness","volume":"69","author":"T Gneiting","year":"2007","journal-title":"J R Stat Soc B Stat Methodology"},{"issue":"4","key":"pcbi.1008618.ref013","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1111\/j.1541-0420.2009.01191.x","article-title":"Predictive model assessment for count data","volume":"65","author":"C Czado","year":"2009","journal-title":"Biometrics"},{"issue":"3","key":"pcbi.1008618.ref014","doi-asserted-by":"crossref","first-page":"788","DOI":"10.1016\/j.ijforecast.2015.12.004","article-title":"Evaluating predictive count data distributions in retail sales forecasting","volume":"32","author":"S. Kolassa","year":"2016","journal-title":"Int J Forecast"},{"issue":"22","key":"pcbi.1008618.ref015","doi-asserted-by":"crossref","first-page":"3443","DOI":"10.1002\/sim.7363","article-title":"Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture","volume":"36","author":"L Held","year":"2017","journal-title":"Stat Med"},{"issue":"2","key":"pcbi.1008618.ref016","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.1006785","article-title":"Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014\u201315","volume":"15","author":"S Funk","year":"2019","journal-title":"PLoS Comput Biol"},{"issue":"164","key":"pcbi.1008618.ref017","article-title":"A novel sub-epidemic modeling framework for short-term forecasting epidemic waves","volume":"17","author":"G Chowell","year":"2019","journal-title":"BMC Med"},{"key":"pcbi.1008618.ref018","article-title":"Sub-epidemic model forecasts for COVID-19 pandemic spread in the USA and European hotspots, February-May 2020","author":"G Chowell","year":"2020","journal-title":"Preprint, medRxiv"},{"issue":"4","key":"pcbi.1008618.ref019","doi-asserted-by":"crossref","first-page":"1267","DOI":"10.5194\/hess-11-1267-2007","article-title":"Verification tools for probabilistic forecasts of continuous hydrological variables","volume":"11","author":"F Laio","year":"2007","journal-title":"Hydrol Earth Syst Sci Discuss"},{"issue":"3","key":"pcbi.1008618.ref020","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1198\/jbes.2010.08110","article-title":"Comparing density forecasts using threshold- and quantile-weighted scoring rules","volume":"29","author":"T Gneiting","year":"2011","journal-title":"J Bus Econ Stat"},{"issue":"3","key":"pcbi.1008618.ref021","doi-asserted-by":"crossref","first-page":"896","DOI":"10.1016\/j.ijforecast.2016.02.001","article-title":"Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond","volume":"32","author":"T Hong","year":"2016","journal-title":"Int J Forecast."},{"issue":"3","key":"pcbi.1008618.ref022","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1080\/07350015.1995.10524599","article-title":"Comparing predictive accuracy","volume":"13","author":"FX Diebold","year":"1995","journal-title":"J Bus Econ Stat"},{"key":"pcbi.1008618.ref023","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.apenergy.2011.11.004","article-title":"Evaluating the quality of scenarios of short-term wind power generation","volume":"96","author":"P Pinson","year":"2012","journal-title":"Appl Energy"},{"key":"pcbi.1008618.ref024","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.apenergy.2016.05.025","article-title":"Generation and evaluation of space-time trajectories of photovoltaic power","volume":"176","author":"F Golestaneh","year":"2016","journal-title":"Appl Energy"},{"key":"pcbi.1008618.ref025","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11749-008-0114-x","article-title":"Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. With discussion.","volume":"17","author":"T Gneiting","year":"2008","journal-title":"Test"},{"key":"pcbi.1008618.ref026","article-title":"Fixed-time descriptive statistics underestimate extremes of epidemic curve ensembles","author":"JL Juul","year":"2021","journal-title":"Nat Phys"},{"key":"pcbi.1008618.ref027","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02562681","article-title":"Scoring rules and the evaluation of probabilities (with discussion).","volume":"5","author":"RL Winkler","year":"1996","journal-title":"Test"},{"issue":"683","key":"pcbi.1008618.ref028","doi-asserted-by":"crossref","first-page":"1924","DOI":"10.1002\/qj.2284","article-title":"Decomposition and graphical portrayal of the quantile score","volume":"140","author":"S Bentzien","year":"2014","journal-title":"Q J Roy Meteorol Soc"},{"issue":"30","key":"pcbi.1008618.ref029","doi-asserted-by":"crossref","first-page":"4908","DOI":"10.1002\/sim.7488","article-title":"Infectious disease prediction with kernel conditional density estimation","volume":"36","author":"EL Ray","year":"2017","journal-title":"Stat Med"},{"key":"pcbi.1008618.ref030","unstructured":"FluSight Network. Repository cdc-flusight-ensemble. 2020. Accessible online at https:\/\/github.com\/FluSightNetwork\/cdc-flusight-ensemble. Last accessed 13 October 2020."},{"key":"pcbi.1008618.ref031","first-page":"278","article-title":"Statistical theory: The prequential approach. With discussion","volume":"147","author":"AP Dawid","year":"1984","journal-title":"J R Stat Soc A"},{"issue":"494","key":"pcbi.1008618.ref032","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1198\/jasa.2011.r10138","article-title":"Making and evaluating point forecasts","volume":"106","author":"T Gneiting","year":"2011","journal-title":"J Am Stat Assoc"},{"issue":"2","key":"pcbi.1008618.ref033","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1111\/j.2517-6161.1968.tb00740.x","article-title":"A Bayesian approach to calibration","volume":"30","author":"IR Dunsmore","year":"1968","journal-title":"J R Stat Soc B Methodol"},{"issue":"337","key":"pcbi.1008618.ref034","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1080\/01621459.1972.10481224","article-title":"A decision-theoretic approach to interval estimation","volume":"67","author":"RL Winkler","year":"1972","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"pcbi.1008618.ref035","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.ijforecast.2019.04.014","article-title":"The M4 Competition: 100,000 time series and 61 forecasting methods","volume":"36","author":"S Makridakis","year":"2020","journal-title":"Int J Forecast"},{"key":"pcbi.1008618.ref036","unstructured":"M Open Forecasting Center. The M5 Competition: Competitors\u2019 Guide. 2020. Available from: https:\/\/mofc.unic.ac.cy\/wp-content\/uploads\/2020\/03\/M5-Competitors-Guide-Final-10-March-2020.docx (downloaded 12 June 2020)."},{"issue":"6","key":"pcbi.1008618.ref037","doi-asserted-by":"crossref","first-page":"953","DOI":"10.1111\/jtsa.12426","article-title":"On the comparison of interval forecasts","volume":"39","author":"R Askanazi","year":"2018","journal-title":"J Time Ser Anal"},{"key":"pcbi.1008618.ref038","article-title":"Scoring interval forecasts: Equal-tailed, shortest, and modal interval.","author":"JR Brehmer","year":"2021","journal-title":"Bernoulli"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010592","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T00:00:00Z","timestamp":1664928000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008618","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,24]],"date-time":"2024-08-24T03:54:17Z","timestamp":1724471657000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008618"}},"subtitle":[],"editor":[{"given":"Virginia E.","family":"Pitzer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,2,12]]},"references-count":38,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2,12]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1008618","relation":{"correction":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1010592","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,12]]}}}