{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T06:20:00Z","timestamp":1772173200300,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010485","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T00:00:00Z","timestamp":1664928000000}}],"reference-count":66,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T00:00:00Z","timestamp":1663891200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"crossref","award":["R35GM119582"],"award-info":[{"award-number":["R35GM119582"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000030","name":"Centers for Disease Control and Prevention","doi-asserted-by":"publisher","award":["1U01IP001122"],"award-info":[{"award-number":["1U01IP001122"]}],"id":[{"id":"10.13039\/100000030","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single \u201clinear pool\u201d by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010485","type":"journal-article","created":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T14:32:46Z","timestamp":1663943566000},"page":"e1010485","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":16,"title":["An expert judgment model to predict early stages of the COVID-19 pandemic in the United States"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6362-9231","authenticated-orcid":true,"given":"Thomas","family":"McAndrew","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3503-9899","authenticated-orcid":true,"given":"Nicholas G.","family":"Reich","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,9,23]]},"reference":[{"key":"pcbi.1010485.ref001","unstructured":"Coronavirus Disease 2019 (COVID-19);. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/index.html."},{"issue":"13","key":"pcbi.1010485.ref002","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1001\/jama.2020.2648","article-title":"Characteristics of and important lessons from the coronavirus disease 2019 (COVID-19) outbreak in China: summary of a report of 72 314 cases from the Chinese Center for Disease Control and Prevention","volume":"323","author":"Z Wu","year":"2020","journal-title":"Jama"},{"issue":"16","key":"pcbi.1010485.ref003","doi-asserted-by":"crossref","first-page":"1545","DOI":"10.1001\/jama.2020.4031","article-title":"Critical care utilization for the COVID-19 outbreak in Lombardy, Italy: early experience and forecast during an emergency response","volume":"323","author":"G Grasselli","year":"2020","journal-title":"Jama"},{"key":"pcbi.1010485.ref004","unstructured":"Novel Coronavirus (2019-nCoV) situation reports;. https:\/\/www.who.int\/emergencies\/diseases\/novel-coronavirus-2019\/situation-reports."},{"issue":"3","key":"pcbi.1010485.ref005","doi-asserted-by":"crossref","first-page":"e1005248","DOI":"10.1371\/journal.pcbi.1005248","article-title":"A human judgment approach to epidemiological forecasting","volume":"13","author":"DC Farrow","year":"2017","journal-title":"PLoS computational biology"},{"issue":"1","key":"pcbi.1010485.ref006","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-018-36361-9","article-title":"Collaborative efforts to forecast seasonal influenza in the United States, 2015\u20132016","volume":"9","author":"CJ McGowan","year":"2019","journal-title":"Scientific reports"},{"issue":"1","key":"pcbi.1010485.ref007","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12879-018-3124-7","article-title":"Summary results of the 2014-2015 DARPA Chikungunya challenge","volume":"18","author":"SY Del Valle","year":"2018","journal-title":"BMC infectious diseases"},{"issue":"3","key":"pcbi.1010485.ref008","doi-asserted-by":"crossref","first-page":"e0231236","DOI":"10.1371\/journal.pone.0231236","article-title":"Forecasting the novel coronavirus COVID-19","volume":"15","author":"F Petropoulos","year":"2020","journal-title":"PloS one"},{"key":"pcbi.1010485.ref009","article-title":"Estimating the early outbreak cumulative incidence of COVID-19 in the United States: three complementary approaches","author":"FS Lu","year":"2020","journal-title":"medRxiv"},{"key":"pcbi.1010485.ref010","article-title":"Sentinel event surveillance to estimate total SARS-CoV-2 infections, United States","author":"AA Lover","year":"2020","journal-title":"medRxiv"},{"issue":"36","key":"pcbi.1010485.ref011","doi-asserted-by":"crossref","first-page":"22597","DOI":"10.1073\/pnas.2005476117","article-title":"Estimating unobserved SARS-CoV-2 infections in the United States","volume":"117","author":"TA Perkins","year":"2020","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1010485.ref012","author":"NP Jewell","year":"2020","journal-title":"Caution warranted: using the institute for health metrics and evaluation model for predicting the course of the COVID-19 pandemic"},{"key":"pcbi.1010485.ref013","unstructured":"A repository of the data and code used for aggregating expert predictions of COVID19;. https:\/\/github.com\/tomcm39\/COVID19_expert_survey."},{"issue":"2","key":"pcbi.1010485.ref014","doi-asserted-by":"crossref","first-page":"e1514","DOI":"10.1002\/wics.1514","article-title":"Aggregating predictions from experts: A review of statistical methods, experiments, and applications","volume":"13","author":"T McAndrew","year":"2021","journal-title":"Wiley Interdisciplinary Reviews: Computational Statistics"},{"issue":"5","key":"pcbi.1010485.ref015","doi-asserted-by":"crossref","first-page":"1106","DOI":"10.1177\/0956797614524255","article-title":"Psychological strategies for winning a geopolitical forecasting tournament","volume":"25","author":"B Mellers","year":"2014","journal-title":"Psychological science"},{"issue":"4","key":"pcbi.1010485.ref016","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1038\/nmeth.2855","article-title":"Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods","volume":"11","author":"SC Warby","year":"2014","journal-title":"Nature methods"},{"key":"pcbi.1010485.ref017","article-title":"The good judgment project: A large scale test of different methods of combining expert predictions","author":"L Ungar","year":"2012","journal-title":"2012 AAAI Fall Symposium Series"},{"key":"pcbi.1010485.ref018","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1109\/CBMS.2006.87","volume-title":"19th IEEE symposium on computer-based medical systems (CBMS\u201906)","author":"TH Cheng","year":"2006"},{"issue":"3-4","key":"pcbi.1010485.ref019","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/0361-3682(84)90016-3","article-title":"Expert vs novice decision making in accounting: A summary","volume":"9","author":"MJ Bouwman","year":"1984","journal-title":"Accounting, Organizations and Society"},{"issue":"3","key":"pcbi.1010485.ref020","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1175\/1520-0493(1984)112<0413:IOFAEO>2.0.CO;2","article-title":"Impacts of feedback and experience on the quality of subjective probability forecasts. comparison of results from the first and second years of the zierikzee experiment","volume":"112","author":"AH Murphy","year":"1984","journal-title":"Monthly Weather Review"},{"issue":"1","key":"pcbi.1010485.ref021","first-page":"41","article-title":"Reliability of subjective probability forecasts of precipitation and temperature","volume":"26","author":"AH Murphy","year":"1977","journal-title":"Journal of the Royal Statistical Society: Series C (Applied Statistics)"},{"issue":"1","key":"pcbi.1010485.ref022","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.eswa.2012.07.032","article-title":"Computational intelligence for heart disease diagnosis: A medical knowledge driven approach","volume":"40","author":"J Nahar","year":"2013","journal-title":"Expert Systems with Applications"},{"issue":"4157","key":"pcbi.1010485.ref023","doi-asserted-by":"crossref","first-page":"1124","DOI":"10.1126\/science.185.4157.1124","article-title":"Judgment under uncertainty: Heuristics and biases","volume":"185","author":"A Tversky","year":"1974","journal-title":"science"},{"key":"pcbi.1010485.ref024","doi-asserted-by":"crossref","DOI":"10.1037\/11281-000","volume-title":"Clinical versus statistical prediction: A theoretical analysis and a review of the evidence","author":"PE Meehl","year":"1954"},{"key":"pcbi.1010485.ref025","first-page":"24I","volume-title":"Regulation of risk: a psychological perspective, tn Regulatory Poliql and the Social Sciences","author":"P Slovic","year":"1985"},{"key":"pcbi.1010485.ref026","volume-title":"The conceptual framework of psychology.(Int. Encycl. unified Sci., v. 1, no. 10.)","author":"E Brunswik","year":"1952"},{"key":"pcbi.1010485.ref027","doi-asserted-by":"crossref","DOI":"10.1525\/9780520350519","volume-title":"Perception and the representative design of psychological experiments","author":"E Brunswik","year":"1956"},{"key":"pcbi.1010485.ref028","volume-title":"Social judgment theory","author":"KR Hammond","year":"1986"},{"issue":"2-3","key":"pcbi.1010485.ref029","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1080\/135467896394474","article-title":"Social judgement theory","volume":"2","author":"ME Doherty","year":"1996","journal-title":"Thinking & Reasoning"},{"issue":"1","key":"pcbi.1010485.ref030","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1207\/s15326969eco1701_2","article-title":"Gaze patterns in the visual control of straight-road driving and braking as a function of speed and expertise","volume":"17","author":"SD Rogers","year":"2005","journal-title":"Ecological Psychology"},{"issue":"1","key":"pcbi.1010485.ref031","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1080\/10407410709336951","article-title":"Ecological validity, representative design, and correspondence between experimental task constraints and behavioral setting: Comment on","volume":"19","author":"D Araujo","year":"2007","journal-title":"Ecological Psychology"},{"key":"pcbi.1010485.ref032","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/S0079-6123(09)01313-2","article-title":"A multiple-cue learning approach as the basis for understanding and improving soccer referees\u2019 decision making","volume":"174","author":"H Plessner","year":"2009","journal-title":"Progress in brain research"},{"key":"pcbi.1010485.ref033","first-page":"209","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","author":"MW Wiggins","year":"2020"},{"issue":"4","key":"pcbi.1010485.ref034","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1016\/0030-5073(70)90024-3","article-title":"Inference behavior in a situation where the cues are not reliably perceived","volume":"5","author":"B Brehmer","year":"1970","journal-title":"Organizational Behavior and Human Performance"},{"issue":"1","key":"pcbi.1010485.ref035","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/0030-5073(73)90003-2","article-title":"Effects of task predictability and cue validity on interpersonal learning of inference tasks involving both linear and nonlinear relations","volume":"10","author":"B Brehmer","year":"1973","journal-title":"Organizational behavior and human performance"},{"issue":"6","key":"pcbi.1010485.ref036","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1037\/0033-2909.83.6.985","article-title":"Social judgment theory and the analysis of interpersonal conflict","volume":"83","author":"B Brehmer","year":"1976","journal-title":"Psychological bulletin"},{"issue":"3","key":"pcbi.1010485.ref037","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1037\/0033-2909.134.3.404","article-title":"Determinants of linear judgment: A meta-analysis of lens model studies","volume":"134","author":"N Karelaia","year":"2008","journal-title":"Psychological bulletin"},{"key":"pcbi.1010485.ref038","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1007\/978-1-4613-2391-4_2","volume-title":"Behavioral decision making","author":"A Tversky","year":"1985"},{"issue":"3","key":"pcbi.1010485.ref039","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1037\/h0048070","article-title":"Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores","volume":"52","author":"PE Meehl","year":"1955","journal-title":"Psychological bulletin"},{"issue":"2","key":"pcbi.1010485.ref040","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/S1364-6613(97)01014-0","article-title":"Confidence in judgment","volume":"1","author":"N Harvey","year":"1997","journal-title":"Trends in cognitive sciences"},{"key":"pcbi.1010485.ref041","author":"IE Dror","year":"2018","journal-title":"When expert decision making goes wrong: Consensus, bias, the role of experts, and accuracy"},{"issue":"2","key":"pcbi.1010485.ref042","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1539-6924.1999.tb00399.x","article-title":"Combining probability distributions from experts in risk analysis","volume":"19","author":"RT Clemen","year":"1999","journal-title":"Risk analysis"},{"issue":"3","key":"pcbi.1010485.ref043","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1016\/S0001-6918(00)00037-8","article-title":"Confidence in aggregation of expert opinions","volume":"104","author":"DV Budescu","year":"2000","journal-title":"Acta psychologica"},{"issue":"5","key":"pcbi.1010485.ref044","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1016\/j.ress.2007.03.005","article-title":"TU Delft expert judgment data base","volume":"93","author":"RM Cooke","year":"2008","journal-title":"Reliability Engineering & System Safety"},{"issue":"6491","key":"pcbi.1010485.ref045","doi-asserted-by":"crossref","first-page":"577","DOI":"10.1126\/science.abb9934","article-title":"Harnessing multiple models for outbreak management","volume":"368","author":"K Shea","year":"2020","journal-title":"Science"},{"issue":"7","key":"pcbi.1010485.ref046","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1038\/s41557-020-0496-2","article-title":"Crowdsourcing drug discovery for pandemics","volume":"12","author":"J Chodera","year":"2020","journal-title":"Nature Chemistry"},{"issue":"4","key":"pcbi.1010485.ref047","doi-asserted-by":"crossref","first-page":"e201","DOI":"10.1016\/S2589-7500(20)30026-1","article-title":"Early epidemiological analysis of the coronavirus disease 2019 outbreak based on crowdsourced data: a population-level observational study","volume":"2","author":"K Sun","year":"2020","journal-title":"The Lancet Digital Health"},{"issue":"5","key":"pcbi.1010485.ref048","doi-asserted-by":"crossref","first-page":"e0250935","DOI":"10.1371\/journal.pone.0250935","article-title":"How well did experts and laypeople forecast the size of the COVID-19 pandemic?","volume":"16","author":"G Recchia","year":"2021","journal-title":"PloS one"},{"issue":"1","key":"pcbi.1010485.ref049","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1002\/for.3980090106","article-title":"Allocating the weights in the linear opinion pool","volume":"9","author":"C Genest","year":"1990","journal-title":"Journal of Forecasting"},{"issue":"2","key":"pcbi.1010485.ref050","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/BF02289503","article-title":"Admissible probability measurement procedures","volume":"31","author":"EH Shuford","year":"1966","journal-title":"Psychometrika"},{"issue":"9","key":"pcbi.1010485.ref051","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1073\/pnas.42.9.654","article-title":"Measures of the value of information","volume":"42","author":"J McCarthy","year":"1956","journal-title":"Proceedings of the National Academy of Sciences of the United States of America"},{"issue":"477","key":"pcbi.1010485.ref052","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"T Gneiting","year":"2007","journal-title":"Journal of the American statistical Association"},{"issue":"8","key":"pcbi.1010485.ref053","doi-asserted-by":"crossref","first-page":"3146","DOI":"10.1073\/pnas.1812594116","article-title":"A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States","volume":"116","author":"NG Reich","year":"2019","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1010485.ref054","first-page":"85","article-title":"The triangular distribution","volume":"9","author":"A Ayyangar","year":"1941","journal-title":"Mathematics Student"},{"key":"pcbi.1010485.ref055","unstructured":"Novel Coronavirus (2019-nCoV) situation reports;. https:\/\/www.cdc.gov\/flu\/about\/burden\/index.html."},{"key":"pcbi.1010485.ref056","unstructured":"I could easily be off 2-fold in either direction, but my best guess is that we\u2019re currently in the 10,000 to 40,000 range nationally. 11\/13;. Twitter."},{"key":"pcbi.1010485.ref057","unstructured":"COVID-19 projections using machine learning;. https:\/\/covid19-projections.com."},{"key":"pcbi.1010485.ref058","article-title":"Bayesian nowcasting with adjustment for delayed and incomplete reporting to estimate COVID-19 infections in the United States","author":"MH Chitwood","year":"2020","journal-title":"medRxiv"},{"issue":"1","key":"pcbi.1010485.ref059","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-021-86811-0","article-title":"A scenario modeling pipeline for COVID-19 emergency planning","volume":"11","author":"JC Lemaitre","year":"2021","journal-title":"Scientific reports"},{"key":"pcbi.1010485.ref060","unstructured":"CEID Covid-19 tracker;. https:\/\/www.covid19.uga.edu\/."},{"key":"pcbi.1010485.ref061","article-title":"Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months","author":"I COVID","year":"2020","journal-title":"MedRxiv"},{"key":"pcbi.1010485.ref062","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780195097344.001.0001","volume-title":"Human judgment and social policy: Irreducible uncertainty, inevitable error, unavoidable injustice","author":"KR Hammond","year":"1996"},{"key":"pcbi.1010485.ref063","volume-title":"Judgments under stress","author":"KR Hammond","year":"2000"},{"key":"pcbi.1010485.ref064","volume-title":"Coherence and correspondence theories in judgment and decision making","author":"KR Hammond","year":"2000"},{"key":"pcbi.1010485.ref065","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-42472-5","volume-title":"Developments in demographic forecasting","author":"S Mazzuco","year":"2020"},{"key":"pcbi.1010485.ref066","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/978-3-030-42472-5_3","volume-title":"Developments in Demographic Forecasting","author":"P Dion","year":"2020"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010485","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T00:00:00Z","timestamp":1664928000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010485","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T11:31:39Z","timestamp":1701084699000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010485"}},"subtitle":[],"editor":[{"given":"Tom","family":"Britton","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,9,23]]},"references-count":66,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,9,23]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010485","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.09.21.20196725","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,23]]}}}