{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T15:20:43Z","timestamp":1780586443973,"version":"3.54.1"},"reference-count":57,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T00:00:00Z","timestamp":1668988800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UO1EB025825"],"award-info":[{"award-number":["UO1EB025825"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006502","name":"Defense Sciences Office, DARPA","doi-asserted-by":"publisher","award":["D20AC00002"],"award-info":[{"award-number":["D20AC00002"]}],"id":[{"id":"10.13039\/100006502","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Immunol."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Preparation to address the critical gap in a future pandemic between non-pharmacological measures and the deployment of new drugs\/vaccines requires addressing two factors: 1) finding virus\/pathogen-agnostic pathophysiological targets to mitigate disease severity and 2) finding a more rational approach to repurposing existing drugs. It is increasingly recognized that acute viral disease severity is heavily driven by the immune response to the infection (\u201ccytokine storm\u201d or \u201ccytokine release syndrome\u201d). There exist numerous clinically available biologics that suppress various pro-inflammatory cytokines\/mediators, but it is extremely difficult to identify clinically effective treatment regimens with these agents. We propose that this is a complex control problem that resists standard methods of developing treatment regimens and accomplishing this goal requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training successful game-playing artificial intelligences (AIs). This proof-of-concept study determines if simulated sepsis (e.g. infection-driven cytokine storm) can be controlled in the absence of effective antimicrobial agents by targeting cytokines for which FDA-approved biologics currently exist.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We use a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. DRL training used a Deep Deterministic Policy Gradient (DDPG) approach with a clinically plausible control interval of 6 hours with manipulation of six cytokines for which there are existing drugs: Tumor Necrosis Factor (TNF), Interleukin-1 (IL-1), Interleukin-4 (IL-4), Interleukin-8 (IL-8), Interleukin-12 (IL-12) and Interferon-\u03b3(IFNg).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>DRL trained an AI policy that could improve outcomes from a baseline Recovered Rate of 61% to one with a Recovered Rate of 90% over ~21 days simulated time. This DRL policy was then tested on four different parameterizations not seen in training representing a range of host and microbe characteristics, demonstrating a range of improvement in Recovered Rate by +33% to +56%<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The current proof-of-concept study demonstrates that significant disease severity mitigation can potentially be accomplished with existing anti-mediator drugs, but only through a multi-modal, adaptive treatment policy requiring implementation with an AI. While the actual clinical implementation of this approach is a projection for the future, the current goal of this work is to inspire the development of a research ecosystem that marries what is needed to improve the simulation models with the development of the sensing\/assay technologies to collect the data needed to iteratively refine those models.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fimmu.2022.995395","type":"journal-article","created":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T02:36:33Z","timestamp":1668998193000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Preparing for the next pandemic: Simulation-based deep reinforcement learning to discover and test multimodal control of systemic inflammation using repurposed immunomodulatory agents"],"prefix":"10.3389","volume":"13","author":[{"given":"Chase","family":"Cockrell","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dale","family":"Larie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gary","family":"An","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,11,21]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.3389\/fimmu.2020.583373","article-title":"The inflammasome in times of COVID-19","volume":"11","author":"de Rivero Vaccari","year":"2020","journal-title":"Front Immunol"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1002\/jmv.26232","article-title":"The cytokine storm and COVID-19","volume":"93","author":"Hu","year":"2021","journal-title":"J Med virology"},{"key":"B3","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1038\/cmi.2015.74","article-title":"The cytokine storm of severe influenza and development of immunomodulatory therapy","volume":"13","author":"Liu","year":"2016","journal-title":"Cell Mol Immunol"},{"key":"B4","doi-asserted-by":"publisher","first-page":"118102","DOI":"10.1016\/j.lfs.2020.118102","article-title":"Contribution of monocytes and macrophages to the local tissue inflammation and cytokine storm in COVID-19: Lessons from SARS and MERS, and potential therapeutic interventions","volume":"257","author":"Jafarzadeh","year":"2020","journal-title":"Life Sci"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1038\/cdd.2015.67","article-title":"Molecular mechanisms of Ebola virus pathogenesis: focus on cell death","volume":"22","author":"Falasca","year":"2015","journal-title":"Cell Death Differentiation"},{"key":"B6","doi-asserted-by":"publisher","first-page":"563\u2013574","DOI":"10.1007\/s00281-017-0625-1","article-title":"Immune-mediated cytokine storm and its role in severe dengue","volume":"39","author":"Srikiatkhachorn","year":"2017","journal-title":"Semin immunopathology"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1080\/08830185.2021.1884248","article-title":"Cytokine release syndrome in COVID-19: a major mechanism of morbidity and mortality","volume":"41","author":"Que","year":"2022","journal-title":"Int Rev Immunol"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1126\/science.abb8925","article-title":"Cytokine release syndrome in severe COVID-19","volume":"368","author":"Moore","year":"2020","journal-title":"Science"},{"key":"B9","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1684\/ecn.2020.0451","article-title":"Cytokine release syndrome: inhibition of pro-inflammatory cytokines as a solution for reducing COVID-19 mortality","volume":"31","author":"Moradian","year":"2020","journal-title":"Eur Cytokine network"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1093\/infdis\/jiaa387","article-title":"The definition and risks of cytokine release syndrome-like in 11 COVID-19-infected pneumonia critically ill patients: disease characteristics and retrospective analysis","volume":"222","author":"Wang","year":"2020"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/S2213-2600(20)30226-5","article-title":"Immunomodulation in COVID-19","volume":"8","author":"Ingraham","year":"2020","journal-title":"Lancet Respir Med"},{"key":"B12","doi-asserted-by":"publisher","first-page":"3419","DOI":"10.2147\/JIR.S322831","article-title":"Immunomodulation as a potent COVID-19 pharmacotherapy: Past, present and future","volume":"14","author":"Hertanto","year":"2021","journal-title":"J Inflammation Res"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1093\/cid\/ciaa1759","article-title":"Immunomodulation as treatment for severe coronavirus disease 2019: A systematic review of current modalities and future directions","volume":"72","author":"Meyerowitz","year":"2021","journal-title":"Clin Infect Dis"},{"key":"B14","doi-asserted-by":"publisher","first-page":"693","DOI":"10.1056\/NEJMoa2021436","article-title":"Dexamethasone in hospitalized patients with covid-19","volume":"384","author":"Horby","year":"2021","journal-title":"N Engl J Med"},{"key":"B15","doi-asserted-by":"publisher","first-page":"517\u2013528","DOI":"10.1007\/s00281-017-0639-8","article-title":"Cytokine storm and sepsis disease pathogenesis","volume":"39","author":"Chousterman","year":"2017","journal-title":"Semin immunopathology"},{"key":"B16","doi-asserted-by":"publisher","first-page":"157-168","DOI":"10.1016\/j.jtbi.2017.07.016","article-title":"Sepsis reconsidered: Identifying novel metrics for behavioral landscape characterization with a high-performance computing implementation of an agent-based model","volume":"430","author":"Cockrell","year":"2017","journal-title":"J Theor Biol"},{"key":"B17","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/978-3-030-56510-7_5","article-title":"Therapeutics as control: Model-based control discovery for sepsis","volume-title":"Complex systems and computational biology approaches to acute inflammation","author":"An","year":"2021"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1097\/01.CCM.0000139707.13729.7D","article-title":"In silico experiments of existing and hypothetical cytokine-directed clinical trials using agent-based modeling","volume":"32","author":"An","year":"2004","journal-title":"Crit Care Med"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1005876","article-title":"Examining the controllability of sepsis using genetic algorithms on an agent-based model of systemic inflammation","volume":"14","author":"Cockrell","year":"2018","journal-title":"PloS Comput Biol"},{"key":"B20","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1089\/cmb.2018.0168","article-title":"Deep reinforcement learning and simulation as a path toward precision medicine","volume":"26","author":"Petersen","year":"2019","journal-title":"J Comput Biol"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1101\/2022.02.17.480940","article-title":"Preparing for the next COVID: Deep reinforcement learning trained artificial intelligence discovery of multi-modal immunomodulatory control of systemic inflammation in the absence of effective anti-microbials","author":"Larie","year":"2022","journal-title":"bioRxiv"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1126\/science.aar6404","article-title":"A general reinforcement learning algorithm that masters chess, shogi, and go through self-play","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z","article-title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning","volume":"575","author":"Vinyals","year":"2019","journal-title":"Nature"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-1051-9","article-title":"An inflammatory cytokine signature predicts COVID-19 severity and survival","volume":"26","author":"Del Valle","year":"2020","journal-title":"Nat Med"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.abd5487","article-title":"Immune responses to SARS-CoV-2 infection in hospitalized pediatric and adult patients","volume":"12","author":"Pierce","year":"2020","journal-title":"Sci Trans Med"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0260623","article-title":"Serial measurement of cytokines strongly predict COVID-19 outcome","volume":"16","author":"Ozger","year":"2021","journal-title":"PLoS One"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.3389\/fimmu.2021.763292","article-title":"Longitudinal cytokine profile in patients with mild to critical COVID-19","volume":"12","author":"Ling","year":"2021","journal-title":"Front Immunol"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2588-y","article-title":"Longitudinal analyses reveal immunological misfiring in severe COVID-19","volume":"584","author":"Lucas","year":"2020","journal-title":"Nature"},{"key":"B30","doi-asserted-by":"publisher","first-page":"e139834","DOI":"10.1172\/jci.insight.139834","article-title":"Longitudinal COVID-19 profiling associates IL-1RA and IL-10 with disease severity and RANTES with mild disease","volume":"5","author":"Zhao","year":"2020","journal-title":"JCI Insight"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1111\/cei.12736","article-title":"Differential evolution of peripheral cytokine levels in symptomatic and asymptomatic responses to experimental influenza virus challenge","volume":"183","author":"McClain","year":"2016","journal-title":"Clin Exp Immunol"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.4049\/jimmunol.172.9.5553","article-title":"Hierarchies in cytokine expression profiles for acute and resolving influenza virus-specific CD8+ T cell responses: correlation of cytokine profile and TCR avidity","volume":"172","author":"La Gruta","year":"2004","journal-title":"J Immunol"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0026050","article-title":"Cytokine response patterns in severe pandemic 2009 H1N1 and seasonal influenza among hospitalized adults","volume":"6","author":"Lee","year":"2011","journal-title":"PLoS One"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0057121","article-title":"The association between serum biomarkers and disease outcome in influenza a (H1N1) pdm09 virus infection: results of two international observational cohort studies","volume":"8","author":"Davey","year":"2013","journal-title":"PLoS One"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcv.2013.05.011","article-title":"Cytokine responses in patients with mild or severe influenza a (H1N1) pdm09","volume":"58","author":"Bradley-Stewart","year":"2013","journal-title":"J Clin Virology"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1089\/omi.2021.0155","article-title":"Mathematical modeling of severe acute respiratory syndrome coronavirus 2 infection network with cytokine storm, oxidative stress, thrombosis, insulin resistance, and nitric oxide pathways","volume":"25","author":"Sasidharakurup","year":"2021","journal-title":"Omics"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1002\/jmv.2586","article-title":"Mathematical modeling of interaction between innate and adaptive immune responses in COVID-19 and implications for viral pathogenesis","volume":"92","author":"Du","year":"2020","journal-title":"J Med virology"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1101\/2020.05.16.097238","article-title":"Mechanistic modeling of the SARS-CoV-2 and immune system interplay unravels design principles for diverse clinicopathological outcomes","author":"Sahoo","year":"2020","journal-title":"bioRxiv"},{"key":"B39","doi-asserted-by":"publisher","first-page":"114863","DOI":"10.1016\/j.molliq.2020.114863","article-title":"Modeling and simulations of CoViD-19 molecular mechanism induced by cytokines storm during SARS-CoV2 infection","volume":"327","author":"Yu","year":"2021","journal-title":"J Mol liquids"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1101\/2022.02.15.480585","article-title":"Mathematical model of a cytokine storm","author":"Kareva","year":"2022","journal-title":"bioRxiv"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.3389\/fmolb.2021.639423","article-title":"A validated mathematical model of the cytokine release syndrome in severe COVID-19","volume":"8","author":"Reis","year":"2021","journal-title":"Front Mol Biosci"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1371\/journal.ppat.1009753","article-title":"COVID-19 virtual patient cohort suggests immune mechanisms driving disease outcomes","volume":"17","author":"Jenner","year":"2021","journal-title":"PloS pathogens"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1008451","article-title":"A modular framework for multiscale, multicellular, spatiotemporal modeling of acute primary viral infection and immune response in epithelial tissues and its application to drug therapy timing and effectiveness","volume":"16","author":"Sego","year":"2020","journal-title":"PLoS Comput Biol"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1101\/2020.04.02.019075","article-title":"Rapid community-driven development of a SARS-CoV-2 tissue simulator","author":"Getz","year":"2020","journal-title":"BioRxiv"},{"key":"B45","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/s11538-016-0225-6","article-title":"Optimization and control of agent-based models in biology: A perspective","volume":"79","author":"An","year":"2017","journal-title":"Bull Math Biol"},{"key":"B46","doi-asserted-by":"publisher","first-page":"150902971","DOI":"10.48550\/arXiv.1509.02971","article-title":"Continuous control with deep reinforcement learning","author":"Lillicrap","year":"2015","journal-title":"arXiv"},{"key":"B47","doi-asserted-by":"publisher","first-page":"13125602","DOI":"10.48550\/arXiv.1312.5602","article-title":"Playing atari with deep reinforcement learning","author":"Mnih","year":"2013","journal-title":"arXiv"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-115-6-457","article-title":"The pathogenesis of sepsis","volume":"115","author":"Bone","year":"1991","journal-title":"Ann Internal Med"},{"key":"B49","doi-asserted-by":"publisher","first-page":"118275","DOI":"10.1016\/j.lfs.2020.118275","article-title":"Utilizing drug repurposing against COVID-19\u2013efficacy, limitations, and challenges","volume":"259","author":"Parvathaneni","year":"2020","journal-title":"Life Sci"},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.3000390","article-title":"Closing the scientific loop: bridging correlation and causality in the petaflop age","volume":"2","author":"An","year":"2010","journal-title":"Sci Trans Med"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(20)30192-8","article-title":"Artificial intelligence in COVID-19 drug repurposing","volume":"2","author":"Zhou","year":"2020","journal-title":"Lancet Digital Health"},{"key":"B52","doi-asserted-by":"publisher","DOI":"10.1016\/j.dsx.2020.06.068","article-title":"Application of artificial intelligence in COVID-19 drug repurposing","volume":"14","author":"Mohanty","year":"2020","journal-title":"Diabetes Metab Syndrome: Clin Res Rev"},{"key":"B53","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1038\/s43588-020-00007-6","article-title":"Lessons from the COVID-19 pandemic for advancing computational drug repurposing strategies","volume":"1","author":"Galindez","year":"2021","journal-title":"Nat Comput Science"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.3389\/fsysb.2022.928387","article-title":"Specialty grand challenge: What it will take to cross the valley of death: Translational systems biology, \u201cTrue\u201d precision medicine, medical digital twins, artificial intelligence and in silico clinical trials","volume":"2","author":"An","year":"2022","journal-title":"Front Syst Biol"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.1177\/0037549720975075","article-title":"Nested active learning for efficient model contextualization and parameterization: pathway to generating simulated populations using multi-scale computational models","volume":"97","author":"Cockrell","year":"2021","journal-title":"Simulation"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.3389\/fphys.2021.662845","article-title":"Utilizing the heterogeneity of clinical data for model refinement and rule discovery through the application of genetic algorithms to calibrate a high-dimensional agent-based model of systemic inflammation","volume":"12","author":"Chase Cockrell","year":"2021","journal-title":"Front Physiol"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.3389\/fsysb.2022.928387","article-title":"Drug development digital twins for drug discovery, testing and repurposing: A schema for requirements and development","volume":"2","author":"An","year":"2022","journal-title":"Front Syst Biol"}],"container-title":["Frontiers in Immunology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fimmu.2022.995395\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T02:36:38Z","timestamp":1668998198000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fimmu.2022.995395\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,21]]},"references-count":57,"alternative-id":["10.3389\/fimmu.2022.995395"],"URL":"https:\/\/doi.org\/10.3389\/fimmu.2022.995395","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.07.25.501428","asserted-by":"object"}]},"ISSN":["1664-3224"],"issn-type":[{"value":"1664-3224","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,21]]},"article-number":"995395"}}