{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T17:44:09Z","timestamp":1793036649251,"version":"4.5.0"},"reference-count":97,"publisher":"National Academy of Sciences","issue":"24","license":[{"start":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T00:00:00Z","timestamp":1749686400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"name":"CDC\u2019s Center for Forecasting and Outbreak Analytics","award":["HHS\/CDC-5U01IP000113"],"award-info":[{"award-number":["HHS\/CDC-5U01IP000113"]}]},{"name":"CDC's Center for Forecasting and Outbreak Analytics","award":["CDC-RFA-FT-23-0069"],"award-info":[{"award-number":["CDC-RFA-FT-23-0069"]}]}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2025,6,17]]},"abstract":"<jats:p>Characterizing the feedback linking human behavior and the transmission of infectious diseases (i.e., behavioral changes) remains a significant challenge in computational and mathematical epidemiology. Existing behavioral epidemic models often lack real-world data calibration and cross-model performance evaluation in both retrospective analysis and forecasting. In this study, we systematically compare the performance of three mechanistic behavioral epidemic models across nine geographies and two modeling tasks during the first wave of COVID-19, using various metrics. The first model, a Data-Driven Behavioral Feedback Model, incorporates behavioral changes by leveraging mobility data to capture variations in contact patterns. The second and third models are Analytical Behavioral Feedback Models, which simulate the feedback loop either through the explicit representation of different behavioral compartments within the population or by utilizing an effective nonlinear force of infection. Our results do not identify a single best model overall, as performance varies based on factors such as data availability, data quality, and the choice of performance metrics. While the Data-Driven Behavioral Feedback Model incorporates substantial real-time behavioral information, the Analytical Compartmental Behavioral Feedback Model often demonstrates superior or equivalent performance in both retrospective fitting and out-of-sample forecasts. Overall, our work offers guidance for future approaches and methodologies to better integrate behavioral changes into the modeling and projection of epidemic dynamics.<\/jats:p>","DOI":"10.1073\/pnas.2421993122","type":"journal-article","created":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T10:49:07Z","timestamp":1749725347000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":30,"title":["Comparative evaluation of behavioral epidemic models using COVID-19 data"],"prefix":"10.1073","volume":"122","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9996-3194","authenticated-orcid":false,"given":"Nicol\u00f2","family":"Gozzi","sequence":"first","affiliation":[{"name":"Institute for Scientific Interchange Foundation","place":["Turin, Italy"]},{"name":"Laboratory for the Modeling of Biological and Socio-technical Systems","place":["Boston"]},{"name":"Northeastern University","place":["Boston"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5559-3064","authenticated-orcid":false,"given":"Nicola","family":"Perra","sequence":"additional","affiliation":[{"name":"Laboratory for the Modeling of Biological and Socio-technical Systems","place":["Boston"]},{"name":"Northeastern University","place":["Boston"]},{"name":"Centre for Complex Systems, School of Mathematical Sciences","place":["London E1 4NS, United Kingdom"]},{"name":"Queen Mary University","place":["London E1 4NS, United Kingdom"]},{"name":"The Alan Turing Institute","place":["London NW1 2DB, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3419-4205","authenticated-orcid":false,"given":"Alessandro","family":"Vespignani","sequence":"additional","affiliation":[{"name":"Institute for Scientific Interchange Foundation","place":["Turin, Italy"]},{"name":"Laboratory for the Modeling of Biological and Socio-technical Systems","place":["Boston"]},{"name":"Northeastern University","place":["Boston"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2025,6,12]]},"reference":[{"key":"e_1_3_4_1_2","first-page":"1","article-title":"Report 2: Estimating the potential total number of novel Coronavirus cases in Wuhan City, China","volume":"22","author":"Imai N.","year":"2020","unstructured":"N. Imai , Report 2: Estimating the potential total number of novel Coronavirus cases in Wuhan City, China. Imp. Coll. Lond. 22 1\u20137 (2020).","journal-title":"Imp. Coll. Lond."},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.3201\/eid2607.200250"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(20)30411-6"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-04130-w"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abc3517"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aba9757"},{"key":"e_1_3_4_7_2","first-page":"1","article-title":"Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand","volume":"16","author":"Ferguson N.M.","year":"2020","unstructured":"N.M. Ferguson , Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID19 mortality and healthcare demand. ICL 16, 1\u201320 (2020).","journal-title":"ICL"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2002616117"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2405-7"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2004978117"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-020-01698-4"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abb4218"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abg3055"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-32404-y"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2021.0900"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1063\/5.0099450"},{"key":"e_1_3_4_17_2","doi-asserted-by":"crossref","unstructured":"M. U. Kraemer Spatiotemporal invasion dynamics of SARS-CoV-2 lineage b. 1.1. 7 emergence. Science 373 889\u2013895 (2021).","DOI":"10.1126\/science.abj0113"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abe6959"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abf1374"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1010146"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijid.2020.12.075"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(21)00057-8"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-021-02190-3"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.physrep.2021.02.001"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s41745-020-00200-6"},{"key":"e_1_3_4_26_2","first-page":"324","article-title":"COVID-19 epidemic prediction and the impact of public health interventions: A review of COVID-19 epidemic models","volume":"6","author":"Xiang Y.","year":"2021","unstructured":"Y. Xiang , COVID-19 epidemic prediction and the impact of public health interventions: A review of COVID-19 epidemic models. Infect. Dis. Model. 6, 324\u2013342 (2021).","journal-title":"Infect. Dis. Model."},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2016.0820"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2014.09.005"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0810762106"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-021-01136-2"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abc0764"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42254-021-00407-1"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1080\/17538947.2021.1952324"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2025.100830"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(20)30165-5"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2293-x"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-22601-6"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(20)30260-9"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.10782"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-020-0931-9"},{"key":"e_1_3_4_41_2","first-page":"100211","article-title":"Understanding COVID-19 dynamics and the effects of interventions in the Philippines: A mathematical modelling study","volume":"14","author":"Caldwell J. M.","year":"2021","unstructured":"J. M. Caldwell , Understanding COVID-19 dynamics and the effects of interventions in the Philippines: A mathematical modelling study. Lancet Reg. Health West. Pac. 14, 100211 (2021).","journal-title":"Lancet Reg. Health West. Pac."},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1009236"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.78933"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-021-00057-5"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2923-3"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.3934\/mbe.2014.11.1337"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0141423"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12879-014-0691-0"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-91514-7"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/2129490"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-89492-x"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2022.104318"},{"key":"e_1_3_4_53_2","first-page":"91","article-title":"Law of mass action and saturation in SIR model with application to Coronavirus modelling","volume":"6","author":"Kolokolnikov T.","year":"2021","unstructured":"T. Kolokolnikov, D. Iron, Law of mass action and saturation in SIR model with application to Coronavirus modelling. Infect. Dis. Model. 6, 91\u201397 (2021).","journal-title":"Infect. Dis. Model."},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1051\/mmnp\/2020011"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2009911117"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.mbs.2020.108441"},{"key":"e_1_3_4_57_2","unstructured":"Google COVID-19 community mobility report (2020). https:\/\/www.google.com\/covid19\/mobility\/. Accessed 1 August 2021."},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2019.100368"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2008.0172"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1515\/9781400841035"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2009.0386"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1093\/aje\/kwt133"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2113561119"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0950268824000347"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.81916"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1010602"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-5474-8"},{"key":"e_1_3_4_68_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1011250108"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1017\/dap.2021.10"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2023.100666"},{"key":"e_1_3_4_71_2","doi-asserted-by":"crossref","unstructured":"S. Milusheva D. Bjorkegren L. Viotti \u201cAssessing bias in smartphone mobility estimates in low income countries\u201d in Proceedings of the 4th ACM SIGCAS Conference on Computing and Sustainable Societies COMPASS \u201921 (Association for Computing Machinery New York NY 2021) pp. 364\u2013378.","DOI":"10.1145\/3460112.3471968"},{"key":"e_1_3_4_72_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-30688-8"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2021.100534"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.plrev.2024.10.011"},{"key":"e_1_3_4_75_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2207537120"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.2105\/AJPH.2022.306831"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abb8001"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2022.100601"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(20)30457-6"},{"key":"e_1_3_4_80_2","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-023-00310-z"},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2016.11.003"},{"key":"e_1_3_4_82_2","first-page":"1801","article-title":"Real-time mechanistic Bayesian forecasts of COVID-19 mortality","volume":"17","author":"Gibson G. C.","year":"2023","unstructured":"G. C. Gibson, N. G. Reich, D. Sheldon, Real-time mechanistic Bayesian forecasts of COVID-19 mortality. AOAS. 17, 1801 (2023).","journal-title":"AOAS"},{"key":"e_1_3_4_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(21)00214-4"},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00523-3"},{"key":"e_1_3_4_85_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(20)30861-6"},{"key":"e_1_3_4_86_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(20)30725-8"},{"key":"e_1_3_4_87_2","doi-asserted-by":"crossref","unstructured":"F. Delussu M. Tizzoni L. Gauvin The limits of human mobility traces to predict the spread of COVID-19: A transfer entropy approach. PNAS nexus 2 pgad302 (2023).","DOI":"10.1093\/pnasnexus\/pgad302"},{"key":"e_1_3_4_88_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1812594116"},{"key":"e_1_3_4_89_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-023-42680-x"},{"key":"e_1_3_4_90_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-20544-y"},{"key":"e_1_3_4_91_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(20)30243-7"},{"key":"e_1_3_4_92_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0023084"},{"key":"e_1_3_4_93_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1002803"},{"key":"e_1_3_4_94_2","doi-asserted-by":"publisher","DOI":"10.21105\/joss.04304"},{"key":"e_1_3_4_95_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1008618"},{"key":"e_1_3_4_96_2","first-page":"10","article-title":"Model selection and multi-model inference","volume":"63","author":"Anderson D.","year":"2004","unstructured":"D. Anderson, K. Burnham, Model selection and multi-model inference. Second. NY Springer Verlag 63, 10 (2004).","journal-title":"Second. NY Springer Verlag"},{"key":"e_1_3_4_97_2","doi-asserted-by":"crossref","unstructured":"N. Gozzi N. Perra A. Vespignani Comparative evaluation of behavioral epidemic models using COVID-19 data. GitHub. https:\/\/github.com\/ngozzi\/covid-behavior-models. Deposited 13 May 2025.","DOI":"10.1101\/2024.11.08.24316998"}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2421993122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,12]],"date-time":"2025-06-12T10:49:16Z","timestamp":1749725356000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2421993122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,12]]},"references-count":97,"journal-issue":{"issue":"24","published-print":{"date-parts":[[2025,6,17]]}},"alternative-id":["10.1073\/pnas.2421993122"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2421993122","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.11.08.24316998","asserted-by":"object"}]},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,12]]},"assertion":[{"value":"2024-10-24","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2421993122"}}