{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T00:53:47Z","timestamp":1793235227801,"version":"4.5.0"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,11,20]],"date-time":"2021-11-20T00:00:00Z","timestamp":1637366400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,20]],"date-time":"2021-11-20T00:00:00Z","timestamp":1637366400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100014895","name":"Open Philanthropy Project","doi-asserted-by":"crossref","award":["Silicon Valley Community Foundation (Grant 165410)"],"award-info":[{"award-number":["Silicon Valley Community Foundation (Grant 165410)"]}],"id":[{"id":"10.13039\/100014895","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Public Health"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>The global spread of COVID-19 has shown that reliable forecasting of public health related outcomes is important but lacking.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We report the results of the first large-scale, long-term experiment in crowd-forecasting of infectious-disease outbreaks, where a total of 562 volunteer participants competed over 15\u2009months to make forecasts on 61 questions with a total of 217 possible answers regarding 19 diseases.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Consistent with the \u201cwisdom of crowds\u201d phenomenon, we found that crowd forecasts aggregated using best-practice adaptive algorithms are well-calibrated, accurate, timely, and outperform all individual forecasters.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Crowd forecasting efforts in public health may be a useful addition to traditional disease surveillance, modeling, and other approaches to evidence-based decision making for infectious disease outbreaks.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12889-021-12083-y","type":"journal-article","created":{"date-parts":[[2021,11,20]],"date-time":"2021-11-20T10:02:49Z","timestamp":1637402569000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Using prediction polling to harness collective intelligence for disease forecasting"],"prefix":"10.1186","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8342-476X","authenticated-orcid":false,"given":"Tara Kirk","family":"Sell","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kelsey Lane","family":"Warmbrod","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Crystal","family":"Watson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc","family":"Trotochaud","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Martin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanjana J.","family":"Ravi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maurice","family":"Balick","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emile","family":"Servan-Schreiber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,11,20]]},"reference":[{"key":"12083_CR1","unstructured":"Harmon K. Advances in disease surveillance: putting the \u201cpublic\u201d into public health. Sci Am. 2010. https:\/\/blogs.scientificamerican.com\/observations\/advances-in-disease-surveillance-putting-the-public-into-public-health\/"},{"key":"12083_CR2","doi-asserted-by":"publisher","unstructured":"Jajosky RA, Groseclose S. Evaluation of reporting timeliness of public health surveillance systems for infectious diseases. BMC Public Health. 2004;4(29, 1). https:\/\/doi.org\/10.1186\/1471-2458-4-29.","DOI":"10.1186\/1471-2458-4-29"},{"key":"12083_CR3","doi-asserted-by":"publisher","unstructured":"Giles J. Wisdom of the crowd. Nature. 2005;438(281, 7066). https:\/\/doi.org\/10.1038\/438281a.","DOI":"10.1038\/438281a"},{"issue":"1","key":"12083_CR4","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1023\/A:1022002107255","volume":"5","author":"J Berg","year":"2003","unstructured":"Berg J, Rietz T. Prediction markets as decision support systems. Inf Syst Front. 2003;5(1):79\u201393. https:\/\/doi.org\/10.1023\/A:1022002107255.","journal-title":"Inf Syst Front"},{"key":"12083_CR5","first-page":"1142","volume":"82","author":"R Forsythe","year":"1992","unstructured":"Forsythe R, Nelson FD, Neumann GR, Wright J. Anatomy of an experimental political stock market. Am Econ Rev. 1992;82:1142\u201361.","journal-title":"Am Econ Rev"},{"key":"12083_CR6","first-page":"69","volume-title":"Laboratory research in political economy","author":"TR Palfrey","year":"1991","unstructured":"Palfrey TR, Forsythe R, Nelson FD, Neumann GR, Wright J. The explanation and prediction of presidential elections: a market alternative to polls. In: Palfrey TR, editor. Laboratory research in political economy. Ann Arbor: University of Michigan Press; 1991. p. 69\u2013112."},{"key":"12083_CR7","doi-asserted-by":"publisher","first-page":"444","DOI":"10.1007\/978-3-642-60495-9","volume-title":"Understanding strategic interaction: essays in honor of Reinhard Selten","author":"W Alberts","year":"1997","unstructured":"Alberts W, Guth W, Hammerstein P, Moldovan B, van Damme E, Forsythe R, et al. What makes markets predict well?. Evidence from the Iowa Electronic Markets. In: Alberts W, Guth W, Hammerstein P, Moldovan B, van Damme E, editors. Understanding strategic interaction: essays in honor of Reinhard Selten. Amsterdam: Springer; 1997. p. 444\u201363."},{"issue":"2","key":"12083_CR8","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1086\/510427","volume":"44","author":"PM Polgreen","year":"2007","unstructured":"Polgreen PM, Nelson FD, Neumann GR. Use of prediction markets to forecast infectious diseases activity. Clin Infect Dis. 2007;44(2):272\u20139. https:\/\/doi.org\/10.1086\/510427.","journal-title":"Clin Infect Dis"},{"issue":"1","key":"12083_CR9","doi-asserted-by":"publisher","first-page":"766","DOI":"10.1186\/s12889-015-2121-7","volume":"15","author":"CY Tung","year":"2015","unstructured":"Tung CY, Chou TC, Lin JW. Using prediction markets of market scoring rule to forecast infectious diseases: a case study in Taiwan. BMC Public Health. 2015;15. https:\/\/bmcpublichealth.biomedcentral.com\/articles\/10.1186\/s12889-015-2121-7. Accessed October 28, 2019(1):766.","journal-title":"BMC Public Health"},{"key":"12083_CR10","unstructured":"Sell TK, Franco C, Ho ATY, Polgreen PM. Using a prediction market to forecast dengue fever activity in the U.S. Emerg Health Threats J. 2011;4:s148."},{"issue":"3","key":"12083_CR11","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1287\/mnsc.2015.2374","volume":"63","author":"P Atanasov","year":"2016","unstructured":"Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, et al. Distilling the wisdom of crowds: prediction markets vs. prediction polls. Manag Sci. 2016;63(3):691\u2013706. https:\/\/doi.org\/10.1287\/mnsc.2015.2374.","journal-title":"Manag Sci"},{"key":"12083_CR12","unstructured":"Program for Monitoring Emerging Diseases (ProMED). https:\/\/promedmail.org\/about-promed\/. Accessed December 10, 2019."},{"key":"12083_CR13","unstructured":"Servan-Schreiber E, And Atanasov, P. Hypermind vs. big data: collective intelligence still dominates electoral forecasting. Collective intelligence 2015, Santa Clara."},{"key":"12083_CR14","unstructured":"Servan-Schreiber E. Debunking three myths about crowd\u2013based forecasting. Collective intelligence 2017, NYU Tandon.\u00a0https:\/\/www.researchgate.net\/publication\/318984024_Debunking_Three_Myths_About_Crowd-Based_Forecasting."},{"key":"12083_CR15","unstructured":"Hypermind use Cases. https:\/\/www.hypermind.com\/en\/case-studies\/. Accessed December 10, 2019."},{"issue":"5","key":"12083_CR16","doi-asserted-by":"publisher","first-page":"1106","DOI":"10.1177\/0956797614524255","volume":"25","author":"B Mellers","year":"2014","unstructured":"Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, et al. Psychological strategies for winning a geopolitical forecasting tournament. Psychol Sci. 2014;25(5):1106\u201315. https:\/\/doi.org\/10.1177\/0956797614524255.","journal-title":"Psychol Sci"},{"key":"12083_CR17","unstructured":"Office of the Director of National intelligence. Aggregative Contingent Estimation (ACE). https:\/\/www.iarpa.gov\/index.php\/research-programs\/ace. Accessed October 30, 2020."},{"key":"12083_CR18","unstructured":"Office of the Director of National intelligence. Hybrid Forecasting Competition (HFC). https:\/\/www.iarpa.gov\/index.php\/research-programs\/hfc. Accessed October 30, 2020."},{"key":"12083_CR19","doi-asserted-by":"publisher","unstructured":"Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78(1):1\u20133. https:\/\/doi.org\/10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2"},{"issue":"4","key":"12083_CR20","doi-asserted-by":"publisher","first-page":"582","DOI":"10.1287\/mnsc.1080.0955","volume":"55","author":"V Jose","year":"2009","unstructured":"Jose V, Nau R, Winkler R. Sensitivity to distance and baseline distributions in forecast evaluation. Manag Sci. 2009;55(4):582\u201390. https:\/\/doi.org\/10.1287\/mnsc.1080.0955.","journal-title":"Manag Sci"},{"issue":"2","key":"12083_CR21","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1287\/deca.2014.0293","volume":"11","author":"J Baron","year":"2014","unstructured":"Baron J, Mellers BA, Tetlock PE, Stone E, Ungar LH. Two reasons to make aggregated probability forecasts more extreme. Decis Anal. 2014;11(2):133\u201345. https:\/\/doi.org\/10.1287\/deca.2014.0293.","journal-title":"Decis Anal"},{"key":"12083_CR22","volume-title":"Comparing forecasting skill and domain expertise for policy-relevant crowd-forecasting. Manuscript in preparation","author":"E Servan-Schreiber","year":"2021","unstructured":"Servan-Schreiber E, Balick M, Ravi SJ, Martin E, Trotochaud M, Watson C, et al. Comparing forecasting skill and domain expertise for policy-relevant crowd-forecasting. Manuscript in preparation; 2021."},{"key":"12083_CR23","doi-asserted-by":"publisher","unstructured":"Murphy AH. A new vector partition of the probability score. J Appl Meteorol. 1973;12(4):595\u2013600. https:\/\/doi.org\/10.1175\/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.","DOI":"10.1175\/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2"}],"container-title":["BMC Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12889-021-12083-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12889-021-12083-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12889-021-12083-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,20]],"date-time":"2021-11-20T22:04:27Z","timestamp":1637445867000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcpublichealth.biomedcentral.com\/articles\/10.1186\/s12889-021-12083-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,20]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["12083"],"URL":"https:\/\/doi.org\/10.1186\/s12889-021-12083-y","relation":{},"ISSN":["1471-2458"],"issn-type":[{"value":"1471-2458","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,20]]},"assertion":[{"value":"3 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Email correspondence with the Johns Hopkins Bloomberg School of Public Health Institutional Review Board concluded that this project did not require an IRB submission. Participants reviewed consent documents and provided electronic written consent to participate during the online signup process.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Emile Servan-Schreiber and Maurice Balick are both partners in Hypermind, llc, the company that provided the crowd-forecasting platform used in this study, as well as 23.5% of the participants.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"2132"}}