{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T18:26:28Z","timestamp":1787682388452,"version":"build-2784847793"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1014644","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T00:00:00Z","timestamp":1787616000000}}],"reference-count":42,"publisher":"Public Library of Science (PLoS)","issue":"8","license":[{"start":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T00:00:00Z","timestamp":1786492800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Epidemic forecasting research often assesses ensembles and their component models using probabilistic scoring rules. Quantifying how individual models affect ensemble performance is challenging, particularly across multiple targets and spatial scales.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We present Winter 2024\u201325 forecasts of Influenza and COVID-19 hospital admissions in England and conduct a retrospective simulation using the operational component models. Forecasts were scored using the per capita weighted interval score (pcWIS) for counts and the ranked probability score (RPS) for ordinal trend direction. We compared retrospective forecasts, used generalised additive models (GAMs) to estimate the expected change in score from the inclusion of a model in a sub-ensemble (an ensemble formed from a subset of available models), and used Pareto analysis to understand which sub-ensembles were Pareto-optimal across scoring rules.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Nationally, there was a 47% improvement in Influenza pcWIS versus sub-ensembles. However, Influenza operational ensembles were on average 22% worse than sub-ensembles, when measured by RPS. For COVID-19, operational ensembles were 43% and 280% worse on average, than retrospective sub-ensembles by pcWIS and RPS, respectively. However, COVID-19 operational ensembles were on average 2% (pcWIS) and 13% (RPS) better than individual operational models. For influenza, operational ensembles were, on average, 58% (pcWIS) and 41% (RPS) better than individual models. The sub-ensemble simulation showed how individual models influenced the ensemble scores during different epidemic phases. The Pareto analysis demonstrated that there can be a trade-off between relative direction and absolute count score optimisation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pcbi.1014644","type":"journal-article","created":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T17:41:53Z","timestamp":1786556513000},"page":"e1014644","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluation of short-term multi-target respiratory forecasts over winter 2024-25 in England using sub-ensemble contribution analyses"],"prefix":"10.1371","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-1522-7088","authenticated-orcid":true,"given":"Jack","family":"Kennedy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Ferguson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8410-9585","authenticated-orcid":true,"given":"Owen","family":"Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven","family":"Riley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Ward","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria L.","family":"Tang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathon","family":"Mellor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,8,12]]},"reference":[{"issue":"2","key":"pcbi.1014644.ref001","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1012849","article-title":"An application of nowcasting methods: Cases of norovirus during the winter 2023\/2024 in England","volume":"21","author":"J Mellor","year":"2025","journal-title":"PLoS Comput Biol"},{"key":"pcbi.1014644.ref002","article-title":"Nowcasting cases and trends during the measles 2023\/24 outbreak in England","author":"ML Tang","year":"2025","journal-title":"medRxiv"},{"issue":"9","key":"pcbi.1014644.ref003","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1011463","article-title":"Nowcasting the 2022 mpox outbreak in England","volume":"19","author":"CE Overton","year":"2023","journal-title":"PLoS Computat Biol"},{"issue":"1","key":"pcbi.1014644.ref004","doi-asserted-by":"crossref","first-page":"6289","DOI":"10.1038\/s41467-024-50601-9","article-title":"Evaluation of FluSight influenza forecasting in the 2021--22 and 2022--23 seasons with a new target laboratory-confirmed influenza hospitalizations","volume":"15","author":"SM Mathis","year":"2024","journal-title":"Nature Commun"},{"key":"pcbi.1014644.ref005","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.81916","article-title":"Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations","volume":"12","author":"K Sherratt","year":"2023","journal-title":"Elife"},{"key":"pcbi.1014644.ref006","article-title":"Performance evaluation of RespiCast ensemble forecasts for primary care syndromic indicators of viral respiratory disease in Europe during the 2023\/24 winter season","author":"N Gozzi","year":"2025","journal-title":"medRxiv"},{"issue":"5","key":"pcbi.1014644.ref007","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1011200","article-title":"Challenges of COVID-19 case forecasting in the US, 2020-2021","volume":"20","author":"V Lopez","year":"2024","journal-title":"PLoS Computat Biol"},{"issue":"2","key":"pcbi.1014644.ref008","article-title":"Prediction of infectious disease epidemics via weighted density ensembles","volume":"14","author":"EL Ray","year":"2018","journal-title":"PLoS Computat Biol"},{"issue":"26","key":"pcbi.1014644.ref009","doi-asserted-by":"crossref","first-page":"4696","DOI":"10.1002\/sim.9884","article-title":"Comparison of combination methods to create calibrated ensemble forecasts for seasonal influenza in the U.S","volume":"42","author":"N Wattanachit","year":"2023","journal-title":"Stat Med"},{"issue":"4","key":"pcbi.1014644.ref010","article-title":"Forecasting COVID-19, influenza, and RSV hospitalizations over winter 2023--4 in England","volume":"53","author":"J Mellor","year":"2025","journal-title":"Int J Epidemiol"},{"key":"pcbi.1014644.ref011","doi-asserted-by":"crossref","first-page":"100765","DOI":"10.1016\/j.epidem.2024.100765","article-title":"Characterising information gains and losses when collecting multiple epidemic model outputs","volume":"47","author":"K Sherratt","year":"2024","journal-title":"Epidemics"},{"key":"pcbi.1014644.ref012","unstructured":"Taylor KS, Taylor JW. A comparison of aggregation methods for probabilistic forecasts of covid-19 mortality in the United States. arXiv preprint arXiv:2007.11103. 2020. https:\/\/arxiv.org\/abs\/2007.11103"},{"issue":"3","key":"pcbi.1014644.ref013","doi-asserted-by":"crossref","first-page":"754","DOI":"10.1016\/j.ijforecast.2015.12.005","article-title":"The forecast combination puzzle: A simple theoretical explanation","volume":"32","author":"G Claeskens","year":"2016","journal-title":"Int J Forecast"},{"issue":"3","key":"pcbi.1014644.ref014","doi-asserted-by":"crossref","first-page":"1366","DOI":"10.1016\/j.ijforecast.2022.06.005","article-title":"Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States","volume":"39","author":"EL Ray","year":"2023","journal-title":"Int J Forecast"},{"issue":"3","key":"pcbi.1014644.ref015","article-title":"Post-processing and weighted combination of infectious disease nowcasts","volume":"21","author":"AVR Amaral","year":"2025","journal-title":"PLoS Computat Biol"},{"issue":"13","key":"pcbi.1014644.ref016","doi-asserted-by":"crossref","first-page":"15647","DOI":"10.1609\/aaai.v37i13.26855","article-title":"Phase-Informed Bayesian Ensemble Models Improve Performance of COVID-19 Forecasts","volume":"37","author":"A Adiga","year":"2023","journal-title":"AAAI"},{"issue":"3","key":"pcbi.1014644.ref017","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1016\/j.ijforecast.2025.12.006","article-title":"Beyond forecast leaderboards: Measuring individual model importance based on contribution to ensemble accuracy","volume":"42","author":"M Kim","year":"2026","journal-title":"Int J Forecast"},{"issue":"1","key":"pcbi.1014644.ref018","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1038\/s43856-023-00424-4","article-title":"Forecasting influenza hospital admissions within English sub-regions using hierarchical generalised additive models","volume":"3","author":"J Mellor","year":"2023","journal-title":"Commun Med (Lond)"},{"issue":"447","key":"pcbi.1014644.ref019","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly Proper Scoring Rules, Prediction, and Estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"pcbi.1014644.ref020","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s12916-024-03369-0","article-title":"Quantifying the impact of hospital catchment area definitions on hospital admissions forecasts: COVID-19 in England, September 2020-April 2021","volume":"22","author":"S Meakin","year":"2024","journal-title":"BMC Med"},{"issue":"8","key":"pcbi.1014644.ref021","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1011393","article-title":"Scoring epidemiological forecasts on transformed scales","volume":"19","author":"NI Bosse","year":"2023","journal-title":"PLoS Computat Biol"},{"issue":"2","key":"pcbi.1014644.ref022","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1008618","article-title":"Evaluating epidemic forecasts in an interval format","volume":"17","author":"J Bracher","year":"2021","journal-title":"PLoS Computat Biol"},{"issue":"6","key":"pcbi.1014644.ref023","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1175\/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2","article-title":"A Scoring System for Probability Forecasts of Ranked Categories","volume":"8","author":"ES Epstein","year":"1969","journal-title":"J Appl Meteor"},{"key":"pcbi.1014644.ref024","unstructured":"Bosse NI, Gruson H, Cori A, van Leeuwen E, Funk S, Abbott S. \u201cEvaluating Forecasts with scoringutils in R,\u201d arXiv preprint arXiv:2205.07090. 2022. https:\/\/arxiv.org\/abs\/2205.07090"},{"key":"pcbi.1014644.ref025","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/978-1-4899-7547-8_5","article-title":"A review on global sensitivity analysis methods","volume-title":"Uncertainty management in simulation-optimization of complex systems: algorithms and applications","author":"B Iooss","year":"2015"},{"issue":"104","key":"pcbi.1014644.ref026","doi-asserted-by":"crossref","first-page":"6962","DOI":"10.21105\/joss.06962","article-title":"gratia: An R package for exploring generalized additive models","volume":"9","author":"GL Simpson","year":"2024","journal-title":"JOSS"},{"key":"pcbi.1014644.ref027","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1016\/j.ins.2014.08.071","article-title":"Methods for multi-objective optimization: An analysis","volume":"293","author":"I Giagkiozis","year":"2015","journal-title":"Inform Sci"},{"key":"pcbi.1014644.ref028","article-title":"Forecastability of infectious disease time series: are some seasons and pathogens intrinsically more difficult to forecast?","author":"LA White","year":"2025","journal-title":"medRxiv"},{"key":"pcbi.1014644.ref029","unstructured":"UK Government Official Statistics. National flu and COVID-19 surveillance reports: 2025 to 2026 season. 2026 [Accessed 21 January 2026]. [Online]. Available: https:\/\/www.gov.uk\/government\/statistics\/national-flu-and-covid-19-surveillance-reports-2025-to-2026-season"},{"issue":"1","key":"pcbi.1014644.ref030","doi-asserted-by":"crossref","first-page":"5379","DOI":"10.1038\/s41467-021-25695-0","article-title":"Trade-offs between individual and ensemble forecasts of an emerging infectious disease","volume":"12","author":"RJ Oidtman","year":"2021","journal-title":"Nat Commun"},{"issue":"57","key":"pcbi.1014644.ref031","doi-asserted-by":"crossref","first-page":"2959","DOI":"10.21105\/joss.02959","article-title":"The targets R package: a dynamic Make-like function-oriented pipeline toolkit for reproducibility and high-performance computing","volume":"6","author":"W Landau","year":"2021","journal-title":"JOSS"},{"key":"pcbi.1014644.ref032","doi-asserted-by":"crossref","first-page":"100379","DOI":"10.1016\/j.epidem.2019.100379","article-title":"Real-time forecasting of epidemic trajectories using computational dynamic ensembles","volume":"30","author":"G Chowell","year":"2020","journal-title":"Epidemics"},{"issue":"1","key":"pcbi.1014644.ref033","doi-asserted-by":"crossref","first-page":"23","DOI":"10.5194\/ascmo-11-23-2025","article-title":"Proper scoring rules for multivariate probabilistic forecasts based on aggregation and transformation","volume":"11","author":"R Pic","year":"2025","journal-title":"Adv Stat Clim Meteorol Oceanogr"},{"key":"pcbi.1014644.ref034","doi-asserted-by":"crossref","first-page":"100819","DOI":"10.1016\/j.epidem.2025.100819","article-title":"Collaborative forecasting of influenza-like illness in Italy: The Influcast experience","volume":"50","author":"S Fiandrino","year":"2025","journal-title":"Epidemics"},{"key":"pcbi.1014644.ref035","doi-asserted-by":"crossref","unstructured":"Adiga A, Hurt B, Kaur G, Lewis B, Marathe M, Porebski P, et al. A Multi-Team Multi-Model Collaborative Covid-19 Forecasting Hub for India. In: 2023 Winter Simulation Conference (WSC). 2023.","DOI":"10.1109\/WSC60868.2023.10407748"},{"key":"pcbi.1014644.ref036","unstructured":"Dudley C, Eisenberg M. Not all accuracy is equal: prioritizing diversity in infectious. 2025."},{"key":"pcbi.1014644.ref037","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.epidem.2016.11.003","article-title":"Real-time forecasting of infectious disease dynamics with a stochastic semi-mechanistic model","volume":"22","author":"S Funk","year":"2018","journal-title":"Epidemics"},{"issue":"4","key":"pcbi.1014644.ref038","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1093\/jrsssa\/qnad030","article-title":"Semi-mechanistic Bayesian modelling of COVID-19 with renewal processes","volume":"186","author":"S Bhatt","year":"2023","journal-title":"J R Stat Soc Ser A: Stat Soc"},{"key":"pcbi.1014644.ref039","unstructured":"Abbott S, Hellewell J, Sherratt K, Gostic K, Hickson J, Badr HS, et al. EpiNow2: Estimate Real-Time Case Counts and Time-Varying Epidemiological Parameters. EpiForecasts. 2025."},{"issue":"9","key":"pcbi.1014644.ref040","first-page":"1967","article-title":"Optimizing disease outbreak forecast ensembles","volume":"30","author":"SJ Fox","year":"2024","journal-title":"Emerg Infect Dis"},{"issue":"1","key":"pcbi.1014644.ref041","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0189988","article-title":"Ensemble method for dengue prediction","volume":"13","author":"AL Buczak","year":"2018","journal-title":"PLoS One"},{"issue":"4","key":"pcbi.1014644.ref042","doi-asserted-by":"crossref","first-page":"2129","DOI":"10.1007\/s10994-023-06465-z","article-title":"Dynamic weighted ensemble for diarrhoea incidence predictions","volume":"113","author":"TD Do","year":"2024","journal-title":"Mach Learn"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1014644","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T00:00:00Z","timestamp":1787616000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1014644","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T17:49:54Z","timestamp":1787680194000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1014644"}},"subtitle":[],"editor":[{"given":"Kimberly M.","family":"Pepin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,8,12]]},"references-count":42,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8,12]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1014644","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,12]]}}}