{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T01:08:36Z","timestamp":1775956116575,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010771","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T00:00:00Z","timestamp":1672272000000}}],"reference-count":25,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Center for Machine Learning and Health at Carnegie Mellon"},{"DOI":"10.13039\/100013925","name":"McCune Charitable Foundation","doi-asserted-by":"publisher","award":["FP00004784"],"award-info":[{"award-number":["FP00004784"]}],"id":[{"id":"10.13039\/100013925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["U01IP001121"],"award-info":[{"award-number":["U01IP001121"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Distributional forecasts are important for a wide variety of applications, including forecasting epidemics. Often, forecasts are miscalibrated, or unreliable in assigning uncertainty to future events. We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations, as well as an extension to make this method more effective in recalibrating epidemic forecasts. This method is guaranteed to improve calibration and log score performance when trained and measured in-sample. We also prove that the increase in expected log score of a recalibrated forecaster is equal to the entropy of the PIT distribution. We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration. This method, available on Github, is effective, robust, and easy to use as a post-processing tool to improve epidemic forecasts.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010771","type":"journal-article","created":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T19:00:52Z","timestamp":1671130852000},"page":"e1010771","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":3,"title":["Recalibrating probabilistic forecasts of epidemics"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9181-1794","authenticated-orcid":true,"given":"Aaron","family":"Rumack","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2158-8304","authenticated-orcid":true,"given":"Ryan J.","family":"Tibshirani","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3274-5862","authenticated-orcid":true,"given":"Roni","family":"Rosenfeld","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,12,15]]},"reference":[{"issue":"643","key":"pcbi.1010771.ref001","doi-asserted-by":"crossref","first-page":"1512","DOI":"10.1002\/qj.456","article-title":"Reliability, sufficiency, and the decomposition of proper scores","volume":"135","author":"J Br\u00f6cker","year":"2009","journal-title":"Quarterly Journal of the Royal Meteorological Society"},{"issue":"4","key":"pcbi.1010771.ref002","first-page":"1251","article-title":"Calibration-based empirical probability","volume":"13","author":"AP Dawid","year":"1985","journal-title":"Annals of Statistics"},{"key":"pcbi.1010771.ref003","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/j.1467-9868.2007.00587.x","article-title":"Probabilistic forecasts, calibration and sharpness","volume":"69","author":"T Gneiting","year":"2007","journal-title":"Journal of the Royal Statistical Society, Series B"},{"issue":"477","key":"pcbi.1010771.ref004","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"Journal of the American Statistical Association"},{"issue":"5","key":"pcbi.1010771.ref005","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1287\/mnsc.1040.0205","article-title":"Probability judgements for continuous quantities","volume":"50","author":"SC Hora","year":"2004","journal-title":"Management Science"},{"key":"pcbi.1010771.ref006","doi-asserted-by":"crossref","first-page":"1312","DOI":"10.1175\/1520-0493(1997)125<1312:VOERSR>2.0.CO;2","article-title":"Verification of Eta-RSM short-range ensemble forecasts","volume":"125","author":"TM Hamill","year":"1997","journal-title":"Monthly Weather Review"},{"key":"pcbi.1010771.ref007","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.1175\/MWR2906.1","article-title":"Using Bayesian model averaging to calibrate forecast ensembles","volume":"133","author":"AE Raftery","year":"2005","journal-title":"Monthly Weather Review"},{"key":"pcbi.1010771.ref008","doi-asserted-by":"crossref","first-page":"1747","DOI":"10.1214\/13-EJS823","article-title":"Combining predictive distributions","volume":"7","author":"T Gneiting","year":"2013","journal-title":"Electronic Journal of Statistics"},{"key":"pcbi.1010771.ref009","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1175\/WAF-D-16-0115.1","article-title":"The probability anomaly correlation and calibration of probabilistic forecasts","volume":"32","author":"H van den Dool","year":"2017","journal-title":"Weather and Forecasting"},{"issue":"4","key":"pcbi.1010771.ref010","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1109\/32.54297","article-title":"Recalibrating software reliability models","volume":"16","author":"S Brocklehurst","year":"1990","journal-title":"IEEE Transactions on Software Engineering"},{"key":"pcbi.1010771.ref011","doi-asserted-by":"crossref","first-page":"1434","DOI":"10.1175\/1520-0493(2004)132<1434:ERIMFS>2.0.CO;2","article-title":"Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts","volume":"132","author":"TM Hamill","year":"2004","journal-title":"Monthly Weather Review"},{"key":"pcbi.1010771.ref012","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1175\/MWR2904.1","article-title":"Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation","volume":"133","author":"T Gneiting","year":"2005","journal-title":"Monthly Weather Review"},{"key":"pcbi.1010771.ref013","doi-asserted-by":"crossref","first-page":"2379","DOI":"10.1175\/MWR3402.1","article-title":"Comparison of ensemble-MOS methods using GFS reforecasts","volume":"135","author":"DS Wilks","year":"2007","journal-title":"Monthly Weather Review"},{"key":"pcbi.1010771.ref014","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1175\/BAMS-87-1-33","article-title":"Reforecasts: An important dataset for improving weather predictions","volume":"87","author":"TM Hamill","year":"2006","journal-title":"Bulletin of the American Meteorological Society"},{"key":"pcbi.1010771.ref015","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1111\/j.2044-8317.1965.tb00695.x","article-title":"Methods for Discriminating Levels of Partial Knowledge Concerning a Test Item","volume":"18","author":"B de Finetti","year":"1965","journal-title":"The British Journal of Mathematical and Statistical Psychology"},{"key":"pcbi.1010771.ref016","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1111\/j.2517-6161.1952.tb00104.x","article-title":"Rational decisions","volume":"14","author":"IJ Good","year":"1952","journal-title":"Journal of the Royal Statistical Society"},{"key":"pcbi.1010771.ref017","first-page":"278","article-title":"Statistical theory: The prequential approach (with discussion)","volume":"147","author":"AP Dawid","year":"1984","journal-title":"Journal of the Royal Statistical Society Series A"},{"key":"pcbi.1010771.ref018","unstructured":"Centers for Disease Control and Prevention. FluSight: Flu forecasting;. https:\/\/www.cdc.gov\/flu\/weekly\/flusight\/index.html."},{"issue":"11","key":"pcbi.1010771.ref019","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1007486","article-title":"Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S","volume":"15","author":"NG Reich","year":"2019","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1010771.ref020","unstructured":"Centers for Disease Control and Prevention. National, regional, and state level outpatient illness and viral surveillance;. https:\/\/gis.cdc.gov\/grasp\/fluview\/fluportaldashboard.html."},{"key":"pcbi.1010771.ref021","unstructured":"Rumack A, Brooks LC, Hyun S, Tibshirani RJ, Rosenfeld R. Recalibration repository;. https:\/\/github.com\/rumackaaron\/recalibration."},{"issue":"1","key":"pcbi.1010771.ref022","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1111\/j.1467-9868.2009.00726.x","article-title":"Combining probability forecasts","volume":"72","author":"R Ranjan","year":"2010","journal-title":"Journal of the Royal Statistical Society, Series B"},{"key":"pcbi.1010771.ref023","unstructured":"Pan American Health Organization. PAHO\/WHO Data\u2014Dengue Cases;. https:\/\/www3.paho.org\/data\/index.php\/en\/mnu-topics\/indicadores-dengue-en\/dengue-nacional-en\/252-dengue-pais-ano-en.html."},{"key":"pcbi.1010771.ref024","unstructured":"Centers for Disease Control and Prevention. Epidemic Prediction Initiative: Aedes Forecasting;. https:\/\/predict.cdc.gov\/post\/5e8e21ebcd1fbb050eacaa1e."},{"key":"pcbi.1010771.ref025","article-title":"Forecast Intervals for Infectious Disease Models","author":"R Picard","year":"2022","journal-title":"medRxiv"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010771","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T00:00:00Z","timestamp":1672272000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010771","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T14:23:44Z","timestamp":1728570224000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010771"}},"subtitle":[],"editor":[{"given":"Cecile","family":"Viboud","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,12,15]]},"references-count":25,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12,15]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010771","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1010771","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,15]]}}}