{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T06:16:48Z","timestamp":1772173008933,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1009334","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T00:00:00Z","timestamp":1632096000000}}],"reference-count":61,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2021,9,8]],"date-time":"2021-09-08T00:00:00Z","timestamp":1631059200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006754","name":"Army Research Laboratory","doi-asserted-by":"publisher","award":["W911NF-15-1-0562"],"award-info":[{"award-number":["W911NF-15-1-0562"]}],"id":[{"id":"10.13039\/100006754","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Epidemiological models can provide the dynamic evolution of a pandemic but they are based on many assumptions and parameters that have to be adjusted over the time the pandemic lasts. However, often the available data are not sufficient to identify the model parameters and hence infer the unobserved dynamics. Here, we develop a general framework for building a trustworthy data-driven epidemiological model, consisting of a workflow that integrates data acquisition and event timeline, model development, identifiability analysis, sensitivity analysis, model calibration, model robustness analysis, and projection with uncertainties in different scenarios. In particular, we apply this framework to propose a modified susceptible\u2013exposed\u2013infectious\u2013recovered (SEIR) model, including new compartments and model vaccination in order to project the transmission dynamics of COVID-19 in New York City (NYC). We find that we can uniquely estimate the model parameters and accurately project the daily new infection cases, hospitalizations, and deaths, in agreement with the available data from NYC\u2019s government\u2019s website. In addition, we employ the calibrated data-driven model to study the effects of vaccination and timing of reopening indoor dining in NYC.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1009334","type":"journal-article","created":{"date-parts":[[2021,9,8]],"date-time":"2021-09-08T13:31:13Z","timestamp":1631107873000},"page":"e1009334","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":34,"title":["An integrated framework for building trustworthy data-driven epidemiological models: Application to the COVID-19 outbreak in New York City"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1539-0454","authenticated-orcid":true,"given":"Sheng","family":"Zhang","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5158-3272","authenticated-orcid":true,"given":"Joan","family":"Ponce","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4016-300X","authenticated-orcid":true,"given":"Zhen","family":"Zhang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0976-1987","authenticated-orcid":true,"given":"Guang","family":"Lin","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9713-7120","authenticated-orcid":true,"given":"George","family":"Karniadakis","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2021,9,8]]},"reference":[{"key":"pcbi.1009334.ref001","unstructured":"Zheng W. Total Variation Regularization for Compartmental Epidemic Models with Time-varying Dynamics. arXiv preprint arXiv:200400412. 2020; p. 1\u201311."},{"key":"pcbi.1009334.ref002","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/j.idm.2020.03.001","article-title":"Why is it difficult to accurately predict the COVID-19 epidemic?","volume":"5","author":"WC Roda","year":"2020","journal-title":"Infectious Disease Modelling"},{"key":"pcbi.1009334.ref003","first-page":"1","article-title":"Fundamental principles of epidemic spread highlight the immediate need for large-scale serological surveys to assess the stage of the SARS-CoV-2 epidemic","author":"J Lourenco","year":"2020","journal-title":"MedRxiv"},{"issue":"6492","key":"pcbi.1009334.ref004","doi-asserted-by":"crossref","first-page":"742","DOI":"10.1126\/science.abb4557","article-title":"Effective containment explains subexponential growth in recent confirmed COVID-19 cases in China","volume":"368","author":"BF Maier","year":"2020","journal-title":"Science"},{"key":"pcbi.1009334.ref005","first-page":"1","article-title":"Modelling the COVID-19 epidemic and implementation of population-wide interventions in Italy","author":"G Giordano","year":"2020","journal-title":"Nature Medicine"},{"key":"pcbi.1009334.ref006","doi-asserted-by":"crossref","unstructured":"Gaeta G. A simple SIR model with a large set of asymptomatic infectives. arXiv preprint arXiv:200308720. 2020; p. 1\u201323.","DOI":"10.3934\/mine.2021013"},{"key":"pcbi.1009334.ref007","first-page":"1","article-title":"SEIR transmission dynamics model of 2019 nCoV coronavirus with considering the weak infectious ability and changes in latency duration","author":"P Shi","year":"2020","journal-title":"MedRxiv"},{"key":"pcbi.1009334.ref008","article-title":"Research about the optimal strategies for prevention and control of varicella outbreak in a school in a central city of China: Based on an SEIR dynamic model","volume":"148","author":"WT Zha","year":"2020","journal-title":"Epidemiology & Infection"},{"issue":"3","key":"pcbi.1009334.ref009","doi-asserted-by":"crossref","first-page":"1667","DOI":"10.1007\/s11071-020-05743-y","article-title":"SEIR modeling of the COVID-19 and its dynamics","volume":"101","author":"S He","year":"2020","journal-title":"Nonlinear Dynamics"},{"key":"pcbi.1009334.ref010","doi-asserted-by":"crossref","first-page":"103746","DOI":"10.1016\/j.rinp.2020.103746","article-title":"A modified SEIR model to predict the COVID-19 outbreak in Spain and Italy: simulating control scenarios and multi-scale epidemics","volume":"21","author":"L L\u00f3pez","year":"2021","journal-title":"Results in Physics"},{"issue":"3","key":"pcbi.1009334.ref011","doi-asserted-by":"crossref","first-page":"165","DOI":"10.21037\/jtd.2020.02.64","article-title":"Modified SEIR and AI prediction of the epidemics trend of COVID-19 in China under public health interventions","volume":"12","author":"Z Yang","year":"2020","journal-title":"Journal of Thoracic Disease"},{"issue":"2-3","key":"pcbi.1009334.ref012","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.biosystems.2005.06.016","article-title":"A hybrid approach for efficient and robust parameter estimation in biochemical pathways","volume":"83","author":"M Rodriguez-Fernandez","year":"2006","journal-title":"Biosystems"},{"issue":"1","key":"pcbi.1009334.ref013","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1186\/1471-2105-7-483","article-title":"Novel metaheuristic for parameter estimation in nonlinear dynamic biological systems","volume":"7","author":"M Rodriguez-Fernandez","year":"2006","journal-title":"BMC bioinformatics"},{"key":"pcbi.1009334.ref014","first-page":"1","article-title":"Structural identifiability and observability of compartmental models of the COVID-19 pandemic","author":"G Massonis","year":"2020","journal-title":"Annual reviews in control"},{"key":"pcbi.1009334.ref015","unstructured":"Gallo L, Frasca M, Latora V, Russo G. Lack of practical identifiability may hamper reliable predictions in COVID-19 epidemic models. arXiv preprint arXiv:201200443. 2020; p. 1\u201333."},{"key":"pcbi.1009334.ref016","doi-asserted-by":"crossref","first-page":"110090","DOI":"10.1016\/j.chaos.2020.110090","article-title":"The susceptible-unidentified infected-confirmed (SUC) epidemic model for estimating unidentified infected population for COVID-19","volume":"139","author":"C Lee","year":"2020","journal-title":"Chaos, Solitons & Fractals"},{"key":"pcbi.1009334.ref017","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.mbs.2018.02.004","article-title":"Structural and practical identifiability analysis of outbreak models","volume":"299","author":"N Tuncer","year":"2018","journal-title":"Mathematical Biosciences"},{"issue":"1-3","key":"pcbi.1009334.ref018","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/S0378-4754(00)00270-6","article-title":"Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates","volume":"55","author":"IM Sobol","year":"2001","journal-title":"Mathematics and computers in simulation"},{"key":"pcbi.1009334.ref019","unstructured":"WHO. Coronavirus disease 2019 (COVID-19) Situation Report\u201451; 2020. https:\/\/apps.who.int\/iris\/bitstream\/handle\/10665\/331475\/nCoVsitrep11Mar2020-eng.pdf?sequence=1&isAllowed=y."},{"key":"pcbi.1009334.ref020","unstructured":"Yan H, Sgueglia K. New York\u2019s first case of coronavirus is a health care worker, and officials say more cases are \u2018inevitable\u2019; 2020. Available from: https:\/\/edition.cnn.com\/2020\/03\/02\/us\/new-york-coronavirus-first-case\/index.htm."},{"key":"pcbi.1009334.ref021","unstructured":"West MG. First Case of Coronavirus Confirmed in New York State; 2020. Available from: https:\/\/www.wsj.com\/articles\/first-case-of-coronavirus-confirmed-in-new-york-state-11583111692."},{"key":"pcbi.1009334.ref022","unstructured":"Governor A M Cuomo. Governor Cuomo Signs the\u2019New York State on PAUSE\u2019 Executive Order; 2020. https:\/\/www.governor.ny.gov\/news\/governor-cuomo-signs-new-york-state-pause-executive-order."},{"key":"pcbi.1009334.ref023","unstructured":"NYC Health Department. NYC Coronavirus Disease 2019 (COVID-19) Data; 2020. https:\/\/github.com\/nychealth\/coronavirus-data#nyc-coronavirus-disease-2019-covid-19-data."},{"issue":"4","key":"pcbi.1009334.ref024","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1056\/NEJMp2016822","article-title":"Wrong but useful\u2013what covid-19 epidemiologic models can and cannot tell us","volume":"383","author":"I Holmdahl","year":"2020","journal-title":"New England Journal of Medicine"},{"key":"pcbi.1009334.ref025","unstructured":"Governor A M Cuomo. At Novel Coronavirus Briefing, Governor Cuomo Declares State of Emergency to Contain Spread of Virus; 2020. https:\/\/www.governor.ny.gov\/news\/novel-coronavirus-briefing-governor-cuomo-declares-state-emergency-contain-spread-virus."},{"key":"pcbi.1009334.ref026","unstructured":"Governor A M Cuomo. Amid Ongoing COVID-19 Pandemic, Governor Cuomo Issues Executive Order Requiring All People in New York to Wear Masks or Face Coverings in Public; 2020. https:\/\/www.governor.ny.gov\/news\/amid-ongoing-covid-19-pandemic-governor-cuomo-issues-executive-order-requiring-all-people-new."},{"key":"pcbi.1009334.ref027","unstructured":"CDC. Considerations for Wearing Masks: Help Slow the Spread of COVID-19;2020. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/prevent-getting-sick\/cloth-face-cover-guidance.html(03-April-2020)."},{"key":"pcbi.1009334.ref028","unstructured":"Governor A M Cuomo. NY Forward: A Guide to Reopening New York & Building Back Better; 2020. https:\/\/www.governor.ny.gov\/sites\/governor.ny.gov\/files\/atoms\/files\/NYForwardReopeningGuide.pdf."},{"key":"pcbi.1009334.ref029","unstructured":"New York State Government. Phase One Industry; 2020. https:\/\/forward.ny.gov\/phase-one-industries."},{"key":"pcbi.1009334.ref030","unstructured":"New York State Government. Phase Two Industry; 2020. https:\/\/forward.ny.gov\/phase-two-industries."},{"key":"pcbi.1009334.ref031","unstructured":"Andrew S, Myles M. NYC Indoor Dining to Resume Sept. 30 With Heavy Restrictions, Cuomo Says; 2020. https:\/\/www.nbcnewyork.com\/news\/local\/nyc-casinos-malls-reopen-today-cuomo-and-de-blasio-face-lawsuit-over-indoor-dining\/2607750."},{"key":"pcbi.1009334.ref032","unstructured":"Gold M. Indoor Dining Will Shut Down in New York City Again; 2020. https:\/\/www.nytimes.com\/2020\/12\/11\/nyregion\/indoor-dining-nyc.html."},{"key":"pcbi.1009334.ref033","unstructured":"Luis FS, Joseph G. 1st Vaccination in U.S. Is Given in New York, Hard Hit in Outbreak\u2019s First Days; 2020. https:\/\/www.nytimes.com\/2020\/12\/14\/nyregion\/coronavirus-vaccine-new-york.html."},{"issue":"27","key":"pcbi.1009334.ref034","doi-asserted-by":"crossref","first-page":"2603","DOI":"10.1056\/NEJMoa2034577","article-title":"Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine","volume":"383","author":"FP Polack","year":"2020","journal-title":"New England Journal of Medicine"},{"key":"pcbi.1009334.ref035","first-page":"403","article-title":"Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine","author":"LR Baden","year":"2020","journal-title":"New England Journal of Medicine"},{"key":"pcbi.1009334.ref036","unstructured":"CDC. COVID-19 Pandemic Planning Scenarios; 2020. https:\/\/www.cdc.gov\/coronavirus\/2019-ncov\/hcp\/planning-scenarios.html(10-July-2020)."},{"issue":"7821","key":"pcbi.1009334.ref037","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1038\/s41586-020-2554-8","article-title":"Reconstruction of the full transmission dynamics of COVID-19 in Wuhan","volume":"584","author":"X Hao","year":"2020","journal-title":"Nature"},{"issue":"9","key":"pcbi.1009334.ref038","doi-asserted-by":"crossref","first-page":"1156","DOI":"10.1001\/jamainternmed.2020.2020","article-title":"Contact tracing assessment of COVID-19 transmission dynamics in Taiwan and risk at different exposure periods before and after symptom onset","volume":"180","author":"HY Cheng","year":"2020","journal-title":"JAMA internal medicine"},{"key":"pcbi.1009334.ref039","doi-asserted-by":"crossref","unstructured":"Byambasuren O, Cardona M, Bell K, Clark J, McLaws ML, Glasziou P. Estimating the extent of true asymptomatic COVID-19 and its potential for community transmission: systematic review and meta-analysis. Available at SSRN 3586675. 2020; p. 1\u201314.","DOI":"10.2139\/ssrn.3586675"},{"issue":"10","key":"pcbi.1009334.ref040","doi-asserted-by":"crossref","first-page":"2000180","DOI":"10.2807\/1560-7917.ES.2020.25.10.2000180","article-title":"Estimating the asymptomatic proportion of coronavirus disease 2019 (COVID-19) cases on board the Diamond Princess cruise ship, Yokohama, Japan, 2020","volume":"25","author":"K Mizumoto","year":"2020","journal-title":"Eurosurveillance"},{"issue":"14","key":"pcbi.1009334.ref041","doi-asserted-by":"crossref","first-page":"1406","DOI":"10.1001\/jama.2020.2565","article-title":"Presumed asymptomatic carrier transmission of COVID-19","volume":"323","author":"Y Bai","year":"2020","journal-title":"JAMA"},{"key":"pcbi.1009334.ref042","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.ijid.2020.03.020","article-title":"Estimation of the asymptomatic ratio of novel coronavirus infections (COVID-19)","volume":"94","author":"H Nishiura","year":"2020","journal-title":"International journal of infectious diseases"},{"issue":"10225","key":"pcbi.1009334.ref043","doi-asserted-by":"crossref","first-page":"689","DOI":"10.1016\/S0140-6736(20)30260-9","article-title":"Nowcasting and forecasting the potential domestic and international spread of the 2019-nCoV outbreak originating in Wuhan, China: a modelling study","volume":"395","author":"JT Wu","year":"2020","journal-title":"The Lancet"},{"issue":"7","key":"pcbi.1009334.ref044","doi-asserted-by":"crossref","DOI":"10.3201\/eid2607.201595","article-title":"Evidence supporting transmission of severe acute respiratory syndrome coronavirus 2 while presymptomatic or asymptomatic","volume":"26","author":"NW Furukawa","year":"2020","journal-title":"Emerging infectious diseases"},{"issue":"13","key":"pcbi.1009334.ref045","doi-asserted-by":"crossref","first-page":"377","DOI":"10.15585\/mmwr.mm6913e1","article-title":"Asymptomatic and presymptomatic SARS-CoV-2 infections in residents of a long-term care skilled nursing facility\u2014King County, Washington, March 2020","volume":"69","author":"A Kimball","year":"2020","journal-title":"Morbidity and Mortality Weekly Report"},{"issue":"1054","key":"pcbi.1009334.ref046","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1098\/rstb.1981.0005","article-title":"The population dynamics of microparasites and their invertebrate hosts","volume":"291","author":"RM Anderson","year":"1981","journal-title":"Philosophical Transactions of the Royal Society of London B, Biological Sciences"},{"key":"pcbi.1009334.ref047","first-page":"1","article-title":"Trends in Covid-19 risk-adjusted mortality rates in a single health system","author":"L Horwitz","year":"2020","journal-title":"medRxiv"},{"key":"pcbi.1009334.ref048","first-page":"1","article-title":"Factors associated with hospitalization and critical illness among 4103 patients with COVID-19 disease in New York City","author":"CM Petrilli","year":"2020","journal-title":"MedRxiv"},{"issue":"3-4","key":"pcbi.1009334.ref049","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/0025-5564(70)90132-X","article-title":"On structural identifiability","volume":"7","author":"R Bellman","journal-title":"Mathematical Biosciences"},{"key":"pcbi.1009334.ref050","doi-asserted-by":"crossref","first-page":"577","DOI":"10.2307\/1913267","article-title":"Identification in parametric models","author":"TJ Rothenberg","year":"1971","journal-title":"Econometrica: Journal of the Econometric Society"},{"issue":"6","key":"pcbi.1009334.ref051","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1109\/TAC.1974.1100711","article-title":"Parametrizations of linear dynamical systems: Canonical forms and identifiability","volume":"19","author":"K Glover","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"7","key":"pcbi.1009334.ref052","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1080\/00207727808941738","article-title":"Identifiability of dynamic systems","volume":"9","author":"A Thowsen","year":"1978","journal-title":"International Journal of Systems Science"},{"issue":"2","key":"pcbi.1009334.ref053","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1109\/TAC.1977.1101474","article-title":"Structural identifiability in linear time-invariant systems","volume":"22","author":"J Reid","year":"1977","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"16","key":"pcbi.1009334.ref054","doi-asserted-by":"crossref","first-page":"2873","DOI":"10.1093\/bioinformatics\/bty1069","article-title":"SIAN: software for structural identifiability analysis of ODE models","volume":"35","author":"H Hong","year":"2019","journal-title":"Bioinformatics"},{"issue":"8","key":"pcbi.1009334.ref055","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1093\/bioinformatics\/btx735","article-title":"GenSSI2.0: multi-experiment structural identifiability analysis of SBML models","volume":"34","author":"TS Ligon","year":"2018","journal-title":"Bioinformatics"},{"key":"pcbi.1009334.ref056","unstructured":"Our World in Data Organization Our world in data. 2021. https:\/\/ourworldindata.org\/grapher\/covid-vaccine-doses-by-930manufacturer?tab=table."},{"issue":"1","key":"pcbi.1009334.ref057","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1137\/090757009","article-title":"On identifiability of nonlinear ODE models and applications in viral dynamics","volume":"53","author":"H Miao","year":"2011","journal-title":"SIAM review"},{"key":"pcbi.1009334.ref058","unstructured":"Hospitals NH. NYC Test & Trace Corps; 2021. https:\/\/www.nychealthandhospitals.org\/test-and-trace\/take-care\/."},{"key":"pcbi.1009334.ref059","unstructured":"Patel J, Fernandes G, Anchuri K. SELF-ISOLATION: SUPPORT, MONITORING, AND ADHERENCE: A scoping review of international approaches; 2021. https:\/\/static1.squarespace.com\/static\/93856ebbd6827d4bdff1f7e7ae1\/t\/600ff28547f88841394166bb\/1611657863707\/939Covid+Isolation+Review+GHGP+20012021.pdf."},{"key":"pcbi.1009334.ref060","doi-asserted-by":"crossref","unstructured":"Marago I, Minen I. Hospital-acquired COVID-19 infection\u2013the magnitude of the problem. Available at SSRN 3622387. 2020.","DOI":"10.2139\/ssrn.3622387"},{"issue":"9","key":"pcbi.1009334.ref061","doi-asserted-by":"crossref","first-page":"e2020498","DOI":"10.1001\/jamanetworkopen.2020.20498","article-title":"Incidence of nosocomial COVID-19 in patients hospitalized at a large US academic medical center","volume":"3","author":"C Rhee","year":"2020","journal-title":"JAMA network open"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1009334","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T00:00:00Z","timestamp":1632096000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009334","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,8]],"date-time":"2023-01-08T16:18:02Z","timestamp":1673194682000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009334"}},"subtitle":[],"editor":[{"given":"Sergei L.","family":"Kosakovsky Pond","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2021,9,8]]},"references-count":61,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,9,8]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1009334","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.02.22.21252255","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,8]]}}}