{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T22:41:31Z","timestamp":1787870491033,"version":"build-2784847793"},"reference-count":56,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T00:00:00Z","timestamp":1667433600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"BroadStreet Health"},{"name":"BroadStreet Health"},{"name":"BroadStreet Health"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Importance<\/jats:title>\n                    <jats:p>The rapid proliferation of COVID-19 has left governments scrambling, and several data aggregators are now assisting in the reporting of county cases and deaths. The different variables affecting reporting (e.g., time delays in reporting) necessitates a well-documented reliability study examining the data methods and discussion of possible causes of differences between aggregators.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To statistically evaluate the reliability of COVID-19 data across aggregators using case fatality rate (CFR) estimates and reliability statistics.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Design, setting, and participants<\/jats:title>\n                    <jats:p>Cases and deaths were collected daily by volunteers via state and local health departments, as primary sources and newspaper reports, as secondary sources. In an effort to begin comparison for reliability statistical analysis, BroadStreet collected data from other COVID-19 aggregator sources, including USAFacts, Johns Hopkins University, New York Times, The COVID Tracking Project.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Main outcomes and measures<\/jats:title>\n                    <jats:p>COVID-19 cases and death counts at the county and state levels.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec005\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Lower levels of inter-rater agreement were observed across aggregators associated with the number of deaths, which manifested itself in state level Bayesian estimates of COVID-19 fatality rates.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec006\">\n                    <jats:title>Conclusions and relevance<\/jats:title>\n                    <jats:p>A national, publicly available data set is needed for current and future disease outbreaks and improved reliability in reporting.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0251470","type":"journal-article","created":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T13:41:22Z","timestamp":1667482882000},"page":"e0251470","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":21,"title":["Reliability of COVID-19 data: An evaluation and reflection"],"prefix":"10.1371","volume":"17","author":[{"given":"April R.","family":"Miller","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samin","family":"Charepoo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erik","family":"Yan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryan W.","family":"Frost","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zachary J.","family":"Sturgeon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Grace","family":"Gibbon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2612-993X","authenticated-orcid":true,"given":"Patrick N.","family":"Balius","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cedonia S.","family":"Thomas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Melanie A.","family":"Schmitt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel A.","family":"Sass","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James B.","family":"Walters","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tracy L.","family":"Flood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas A.","family":"Schmitt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"on behalf of the COVID-19 Data Project","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,11,3]]},"reference":[{"issue":"8","key":"pone.0251470.ref001","doi-asserted-by":"crossref","first-page":"e435","DOI":"10.1016\/S2589-7500(20)30142-4","article-title":"Applications of digital technology in COVID-19 pandemic planning and response","volume":"2","author":"S Whitelaw","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"2","key":"pone.0251470.ref002","doi-asserted-by":"crossref","first-page":"100022","DOI":"10.1016\/j.patter.2020.100022","article-title":"COVID-19 Is a Data Science Issue","volume":"1","author":"S. Callaghan","year":"2020","journal-title":"Patterns N Y N"},{"key":"pone.0251470.ref003","unstructured":"Data Collection and Reporting | NNDSS [Internet]. [cited 2021 Mar 12]. https:\/\/wwwn.cdc.gov\/nndss\/data-collection.html."},{"key":"pone.0251470.ref004","unstructured":"Killeen BD, Wu JY, Shah K, Zapaishchykova A, Nikutta P, Tamhane A, et al. A County-level Dataset for Informing the United States\u2019 Response to COVID-19. ArXiv200400756 Phys Q-Bio [Internet]. 2020 Sep 10 [cited 2021 Mar 12]; http:\/\/arxiv.org\/abs\/2004.00756."},{"key":"pone.0251470.ref005","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1186\/s12942-015-0011-y","article-title":"The mortality rates and the space-time patterns of John Snow\u2019s cholera epidemic map","volume":"14","author":"N Shiode","year":"2015","journal-title":"Int J Health Geogr"},{"issue":"19","key":"pone.0251470.ref006","doi-asserted-by":"crossref","first-page":"1908","DOI":"10.1001\/jama.2020.5910","article-title":"Public Health Interventions for COVID-19: Emerging Evidence and Implications for an Evolving Public Health Crisis","volume":"323","author":"DM Hartley","year":"2020","journal-title":"JAMA"},{"issue":"8","key":"pone.0251470.ref007","doi-asserted-by":"crossref","first-page":"1183","DOI":"10.1038\/s41591-020-1011-4","article-title":"Digital technologies in the public-health response to COVID-19","volume":"26","author":"J Budd","year":"2020","journal-title":"Nat Med"},{"issue":"8","key":"pone.0251470.ref008","doi-asserted-by":"crossref","first-page":"e1008117","DOI":"10.1371\/journal.pcbi.1008117","article-title":"Real-time estimation of disease activity in emerging outbreaks using internet search information","volume":"16","author":"EL Aiken","year":"2020","journal-title":"PLOS Comput Biol"},{"key":"pone.0251470.ref009","unstructured":"Social Distancing [Internet]. [cited 2021 Mar 14]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/prevent-getting-sick\/social-distancing.html."},{"issue":"1","key":"pone.0251470.ref010","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/s12942-020-00202-8","article-title":"Geographical tracking and mapping of coronavirus disease COVID-19\/severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) epidemic and associated events around the world: how 21st century GIS technologies are supporting the global fight against outbreaks and epidemics","volume":"19","author":"MN Kamel Boulos","year":"2020","journal-title":"Int J Health Geogr"},{"issue":"10","key":"pone.0251470.ref011","doi-asserted-by":"crossref","first-page":"e0240153","DOI":"10.1371\/journal.pone.0240153","article-title":"Modeling the dynamics of the COVID-19 population in Australia: A probabilistic analysis","volume":"15","author":"A Eshragh","year":"2020","journal-title":"PLOS ONE"},{"issue":"22","key":"pone.0251470.ref012","doi-asserted-by":"crossref","first-page":"e75","DOI":"10.1056\/NEJMp2006740","article-title":"Thinking Globally, Acting Locally\u2014The U.S. Response to Covid-19","volume":"382","author":"RL Haffajee","year":"2020","journal-title":"N Engl J Med"},{"key":"pone.0251470.ref013","unstructured":"CDC activities and initiatives supporting the COVID-19 response and the President\u2019s plan for opening America up again: May 2020 [Internet]. [cited 2021 Mar 12]. https:\/\/stacks.cdc.gov\/view\/cdc\/88478."},{"issue":"8","key":"pone.0251470.ref014","doi-asserted-by":"crossref","first-page":"e425","DOI":"10.1016\/S2589-7500(20)30137-0","article-title":"Digital tools against COVID-19: taxonomy, ethical challenges, and navigation aid","volume":"2","author":"U Gasser","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"7","key":"pone.0251470.ref015","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1038\/s41562-020-0909-7","article-title":"COVID-19 Government Response Event Dataset (CoronaNet v.1.0)","volume":"4","author":"C Cheng","year":"2020","journal-title":"Nat Hum Behav"},{"issue":"10","key":"pone.0251470.ref016","doi-asserted-by":"crossref","first-page":"e0239886","DOI":"10.1371\/journal.pone.0239886","article-title":"Web and phone-based COVID-19 syndromic surveillance in Canada: A cross-sectional study","volume":"15","author":"L Lapointe-Shaw","year":"2020","journal-title":"PLOS ONE"},{"issue":"10","key":"pone.0251470.ref017","doi-asserted-by":"crossref","first-page":"e0239569","DOI":"10.1371\/journal.pone.0239569","article-title":"Estimating the severity of COVID-19: Evidence from the Italian epicenter","volume":"15","author":"P Buonanno","year":"2020","journal-title":"PLOS ONE"},{"issue":"10","key":"pone.0251470.ref018","doi-asserted-by":"crossref","first-page":"e0239304","DOI":"10.1371\/journal.pone.0239304","article-title":"Use of social media big data as a novel HIV surveillance tool in South Africa","volume":"15","author":"A van Heerden","year":"2020","journal-title":"PLOS ONE"},{"key":"pone.0251470.ref019","unstructured":"(COVID-19) in the U.S. Centers for Disease Control and Prevention. [Internet]. CDC. Coronavirus Disease; 2019 [cited 2020 Aug 19]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/cases-updates\/cases-in-us.html."},{"key":"pone.0251470.ref020","article-title":"Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months","author":"IHME COVID-19 health service utilization forecasting team","year":"2020","journal-title":"Infectious Diseases (except HIV\/AIDS)"},{"issue":"14","key":"pone.0251470.ref021","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1001\/jama.2020.3151","article-title":"Response to COVID-19 in Taiwan: Big Data Analytics, New Technology, and Proactive Testing","volume":"323","author":"CJ Wang","year":"2020","journal-title":"JAMA"},{"issue":"0 1","key":"pone.0251470.ref022","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1080\/00031305.2018.1529623","article-title":"The Role of Expert Judgment in Statistical Inference and Evidence-Based Decision-Making","volume":"73","author":"NC Brownstein","year":"2019","journal-title":"Am Stat"},{"issue":"8","key":"pone.0251470.ref023","doi-asserted-by":"crossref","first-page":"818","DOI":"10.2310\/JIM.0b013e3181c2996c","article-title":"Bridging Clinical Investigators and Statisticians: Writing the Statistical Methodology for a Research Proposal","volume":"57","author":"B Adams-Huet","year":"2009","journal-title":"J Investig Med"},{"issue":"6490","key":"pone.0251470.ref024","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1126\/science.abb3221","article-title":"Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (SARS-CoV-2)","volume":"368","author":"R Li","year":"2020","journal-title":"Science"},{"issue":"10","key":"pone.0251470.ref025","doi-asserted-by":"crossref","first-page":"1336","DOI":"10.1001\/jamainternmed.2020.3391","article-title":"Estimation of Excess Deaths Associated With the COVID-19 Pandemic in the United States, March to May 2020","volume":"180","author":"DM Weinberger","year":"2020","journal-title":"JAMA Intern Med"},{"key":"pone.0251470.ref026","unstructured":"BroadStreet Covid19 Data Project [Internet]. [cited 2021 Mar 12]. https:\/\/covid19dataproject.org\/."},{"key":"pone.0251470.ref027","unstructured":"Detailed Methodology and Sources: COVID-19 Data [Internet]. USAFacts. [cited 2021 Mar 12]. https:\/\/usafacts.org\/articles\/detailed-methodology-covid-19-data\/."},{"key":"pone.0251470.ref028","unstructured":"COVID-19 United States Cases by County [Internet]. Johns Hopkins Coronavirus Resource Center. [cited 2021 Mar 12]. https:\/\/coronavirus.jhu.edu\/us-map."},{"key":"pone.0251470.ref029","unstructured":"Coronavirus in the U.S.: Latest Map and Case Count\u2014The New York Times [Internet]. [cited 2021 Mar 12]. https:\/\/www.nytimes.com\/interactive\/2020\/us\/coronavirus-us-cases.html."},{"key":"pone.0251470.ref030","unstructured":"The COVID Tracking Project [Internet]. The COVID Tracking Project. [cited 2021 Mar 12]. https:\/\/covidtracking.com\/."},{"key":"pone.0251470.ref031","unstructured":"Team\u2014covid19dataproject.org [Internet]. 2020 [cited 2021 Aug 26]. https:\/\/covid19dataproject.org\/team-2\/."},{"key":"pone.0251470.ref032","unstructured":"CDC. Coronavirus Disease 2019 Prevention (COVID-19) [Internet]. Centers for Disease Control and Prevention. 2020 [cited 2020 Apr 5]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/index.html."},{"key":"pone.0251470.ref033","unstructured":"CDC. Cases, Data, and Surveillance [Internet]. Centers for Disease Control and Prevention. 2020 [cited 2021 Mar 12]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/cases-updates\/index.html."},{"key":"pone.0251470.ref034","unstructured":"Yang T, Shen K, He S, Li E, Sun P, Chen P, et al. CovidNet: To Bring Data Transparency in the Era of COVID-19. ArXiv200510948 Cs Q-Bio [Internet]. 2020 Jul 20 [cited 2021 Mar 14]; http:\/\/arxiv.org\/abs\/2005.10948."},{"key":"pone.0251470.ref035","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R foundation for Statistical Computing; 2020. [Internet]. [cited 2020 Sep 10]. https:\/\/www.r-project.org\/."},{"key":"pone.0251470.ref036","unstructured":"BroadStreet-Health [Internet]. GitHub. [cited 2021 Mar 19]. https:\/\/github.com\/BroadStreet-Health."},{"issue":"1","key":"pone.0251470.ref037","doi-asserted-by":"crossref","first-page":"23","DOI":"10.20982\/tqmp.08.1.p023","article-title":"Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial","volume":"8","author":"KA Hallgren","year":"2012","journal-title":"Tutor Quant Methods Psychol"},{"issue":"2","key":"pone.0251470.ref038","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.stamet.2008.06.001","article-title":"A note on the linearly weighted kappa coefficient for ordinal scales","volume":"6","author":"S Vanbelle","year":"2009","journal-title":"Stat Methodol"},{"key":"pone.0251470.ref039","article-title":"Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior","author":"DV Cicchetti","journal-title":"Am J Ment Defic"},{"issue":"3","key":"pone.0251470.ref040","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"ML McHugh","year":"2012","journal-title":"Biochem Medica"},{"key":"pone.0251470.ref041","unstructured":"Thacker S. Council of State and Territorial Epidemiologists.: 8."},{"issue":"1","key":"pone.0251470.ref042","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1001\/jama.2020.9536","article-title":"The Importance of Proper Death Certification During the COVID-19 Pandemic","volume":"324","author":"JR Gill","year":"2020","journal-title":"JAMA"},{"key":"pone.0251470.ref043","unstructured":"WAC 246-101-220: [Internet]. [cited 2021 Mar 16]. https:\/\/app.leg.wa.gov\/wac\/default.aspx?cite=246-101-220."},{"key":"pone.0251470.ref044","unstructured":"covid-19labreporting State of Washington [Internet]. State of Washington Department of Health; 2020 [cited 2021 Mar 15]. https:\/\/www.doh.wa.gov\/portals\/1\/documents\/1600\/coronavirus\/20200307-covid-19labreporting.pdf."},{"key":"pone.0251470.ref045","unstructured":"Case Investigations and Contact Tracing FAQ:: Washington State Department of Health [Internet]. [cited 2021 Mar 16]. https:\/\/www.doh.wa.gov\/Emergencies\/COVID19\/CaseInvestigationsandContactTracing\/CaseInvestigationsandContactTracingFAQ."},{"key":"pone.0251470.ref046","unstructured":"Washington Disease Reporting System General Communicable Diseases Instruction Manual [Internet]. Washington Disease Reporting System; 2018 [cited 2021 Mar 15]. https:\/\/www.doh.wa.gov\/Portals\/1\/Documents\/Pubs\/420-227-WDRS-GCDmanual.pdf."},{"key":"pone.0251470.ref047","article-title":"High agreement but low Kappa: I. the problems of two paradoxes","journal-title":"Journal of Clinical Epidemiology"},{"key":"pone.0251470.ref048","article-title":"High agreement but low kappa: II. Resolving the paradoxes","journal-title":"Journal of Clinical Epidemiology"},{"key":"pone.0251470.ref049","unstructured":"Publicly available COVID-19 data for analytics [Internet]. Google Cloud Blog. [cited 2021 Mar 23]. https:\/\/cloud.google.com\/blog\/products\/data-analytics\/publicly-available-covid-19-data-for-analytics\/."},{"key":"pone.0251470.ref050","unstructured":"Area Deprivation Index (ADI)\u2013Marketplace\u2013Google Cloud Platform [Internet]. [cited 2021 Mar 23]. https:\/\/console.cloud.google.com\/marketplace\/product\/broadstreet-public-data\/adi?project=theta-terrain-285112."},{"key":"pone.0251470.ref051","unstructured":"Area Deprivation Index and Vulnerable Populations\u2014How do I find them? [Internet]. [cited 2021 Mar 23]. https:\/\/help.broadstreet.io\/article\/adi\/."},{"key":"pone.0251470.ref052","unstructured":"BroadStreet-Health\/Race-and-Ethnicity-Data [Internet]. BroadStreet-Health; 2021 [cited 2021 Mar 24]. https:\/\/github.com\/BroadStreet-Health\/Race-and-Ethnicity-Data."},{"key":"pone.0251470.ref053","unstructured":"CDC. Increased Risk Factors for Exposure [Internet]. Centers for Disease Control and Prevention. 2020 [cited 2021 Apr 23]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/community\/health-equity\/racial-ethnic-disparities\/increased-risk-exposure.html."},{"key":"pone.0251470.ref054","unstructured":"CDC. Health Equity Considerations & Racial & Ethnic Minority Groups [Internet]. Centers for Disease Control and Prevention. 2020 [cited 2021 Apr 23]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/community\/health-equity\/race-ethnicity.html."},{"key":"pone.0251470.ref055","unstructured":"CDC. Hospitalization and Death by Race\/Ethnicity [Internet]. Centers for Disease Control and Prevention. 2020 [cited 2021 Apr 23]. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/covid-data\/investigations-discovery\/hospitalization-death-by-race-ethnicity.html."},{"key":"pone.0251470.ref056","unstructured":"CDC. COVID Data Tracker [Internet]. Centers for Disease Control and Prevention. 2020 [cited 2021 Apr 23]. https:\/\/covid.cdc.gov\/covid-data-tracker."}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0251470","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T13:42:03Z","timestamp":1667482923000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0251470"}},"subtitle":[],"editor":[{"given":"Jagdish","family":"Khubchandani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,11,3]]},"references-count":56,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,11,3]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0251470","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.04.25.21256069","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,3]]}}}