{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T16:52:24Z","timestamp":1790268744014,"version":"4.1.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011790","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T00:00:00Z","timestamp":1706659200000}}],"reference-count":47,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T00:00:00Z","timestamp":1705622400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1725729"],"award-info":[{"award-number":["1725729"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>One of the challenges in a viral pandemic is the emergence of novel variants with different phenotypical characteristics. An ability to forecast future viral individuals at the sequence level enables advance preparation by characterizing the sequences and closing vulnerabilities in current preventative and therapeutic methods. In this article, we explore, in the context of a viral pandemic, the problem of generating complete instances of undiscovered viral protein sequences, which have a high likelihood of being discovered in the future using protein language models. Current approaches to training these models fit model parameters to a known sequence set, which does not suit pandemic forecasting as future sequences differ from known sequences in some respects. To address this, we develop a novel method, called PandoGen, to train protein language models towards the pandemic protein forecasting task. PandoGen combines techniques such as synthetic data generation, conditional sequence generation, and reward-based learning, enabling the model to forecast future sequences, with a high propensity to spread. Applying our method to modeling the SARS-CoV-2 Spike protein sequence, we find empirically that our model forecasts twice as many novel sequences with five times the case counts compared to a model that is 30\u00d7 larger. Our method forecasts unseen lineages months in advance, whereas models 4\u00d7 and 30\u00d7 larger forecast almost no new lineages. When trained on data available up to a month before the onset of important Variants of Concern, our method consistently forecasts sequences belonging to those variants within tight sequence budgets.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011790","type":"journal-article","created":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T14:01:21Z","timestamp":1705672881000},"page":"e1011790","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":13,"title":["PandoGen: Generating complete instances of future SARS-CoV-2 sequences using Deep Learning"],"prefix":"10.1371","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7042-0244","authenticated-orcid":true,"given":"Anand","family":"Ramachandran","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven S.","family":"Lumetta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3016-0270","authenticated-orcid":true,"given":"Deming","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,1,19]]},"reference":[{"issue":"8","key":"pcbi.1011790.ref001","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1038\/nmeth.3027","article-title":"Deep mutational scanning: a new style of protein science","volume":"11","author":"DM Fowler","year":"2014","journal-title":"Nature methods"},{"issue":"6","key":"pcbi.1011790.ref002","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1016\/j.cell.2023.02.001","article-title":"A pseudovirus system enables deep mutational scanning of the full SARS-CoV-2 spike","volume":"186","author":"B Dadonaite","year":"2023","journal-title":"Cell"},{"key":"pcbi.1011790.ref003","unstructured":"OpenAI. GPT-4 Technical Report; 2023."},{"key":"pcbi.1011790.ref004","unstructured":"Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, et al. Training language models to follow instructions with human feedback; 2022. Available from: https:\/\/arxiv.org\/abs\/2203.02155."},{"issue":"6624","key":"pcbi.1011790.ref005","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1126\/science.abq1158","article-title":"Competition-level code generation with alphacode","volume":"378","author":"Y Li","year":"2022","journal-title":"Science"},{"key":"pcbi.1011790.ref006","article-title":"Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences","author":"A Rives","year":"2019","journal-title":"PNAS"},{"issue":"10","key":"pcbi.1011790.ref007","doi-asserted-by":"crossref","first-page":"7112","DOI":"10.1109\/TPAMI.2021.3095381","article-title":"Prottrans: Toward understanding the language of life through self-supervised learning","volume":"44","author":"A Elnaggar","year":"2021","journal-title":"IEEE transactions on pattern analysis and machine intelligence"},{"issue":"1","key":"pcbi.1011790.ref008","doi-asserted-by":"crossref","first-page":"4348","DOI":"10.1038\/s41467-022-32007-7","article-title":"ProtGPT2 is a deep unsupervised language model for protein design","volume":"13","author":"N Ferruz","year":"2022","journal-title":"Nature communications"},{"issue":"1","key":"pcbi.1011790.ref009","doi-asserted-by":"crossref","first-page":"2403","DOI":"10.1038\/s41467-021-22732-w","article-title":"Protein design and variant prediction using autoregressive generative models","volume":"12","author":"JE Shin","year":"2021","journal-title":"Nature communications"},{"key":"pcbi.1011790.ref010","first-page":"1","article-title":"Large language models generate functional protein sequences across diverse families","author":"A Madani","year":"2023","journal-title":"Nature Biotechnology"},{"issue":"6637","key":"pcbi.1011790.ref011","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Z Lin","year":"2023","journal-title":"Science"},{"key":"pcbi.1011790.ref012","unstructured":"Hesslow D, Zanichelli N, Notin P, Poli I, Marks D. Rita: a study on scaling up generative protein sequence models. arXiv preprint arXiv:220505789. 2022;."},{"key":"pcbi.1011790.ref013","article-title":"ProGen2: exploring the boundaries of protein language models","author":"E Nijkamp","year":"2022","journal-title":"Cell Systems"},{"issue":"6526","key":"pcbi.1011790.ref014","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1126\/science.abd7331","article-title":"Learning the language of viral evolution and escape","volume":"371","author":"B Hie","year":"2021","journal-title":"Science"},{"issue":"633","key":"pcbi.1011790.ref015","doi-asserted-by":"crossref","first-page":"eabk3445","DOI":"10.1126\/scitranslmed.abk3445","article-title":"Predicting the mutational drivers of future SARS-CoV-2 variants of concern","volume":"14","author":"MC Maher","year":"2022","journal-title":"Science Translational Medicine"},{"key":"pcbi.1011790.ref016","article-title":"A deep generative model of the SARS-CoV-2 spike protein predicts future variants","author":"RM Dhodapkar","year":"2023","journal-title":"bioRxiv"},{"key":"pcbi.1011790.ref017","first-page":"2022","article-title":"Running ahead of evolution-AI based simulation for predicting future high-risk SARS-CoV-2 variants","author":"J Chen","year":"2022","journal-title":"bioRxiv"},{"issue":"6599","key":"pcbi.1011790.ref018","doi-asserted-by":"crossref","first-page":"1327","DOI":"10.1126\/science.abm1208","article-title":"Analysis of 6.4 million SARS-CoV-2 genomes identifies mutations associated with fitness","volume":"376","author":"F Obermeyer","year":"2022","journal-title":"Science"},{"issue":"21","key":"pcbi.1011790.ref019","doi-asserted-by":"crossref","first-page":"4008","DOI":"10.1016\/j.cell.2022.08.024","article-title":"Deep mutational learning predicts ACE2 binding and antibody escape to combinatorial mutations in the SARS-CoV-2 receptor-binding domain","volume":"185","author":"JM Taft","year":"2022","journal-title":"Cell"},{"issue":"8","key":"pcbi.1011790.ref020","doi-asserted-by":"crossref","first-page":"2007","DOI":"10.1038\/s41591-023-02483-5","article-title":"Deep-learning-enabled protein\u2013protein interaction analysis for prediction of SARS-CoV-2 infectivity and variant evolution","volume":"29","author":"G Wang","year":"2023","journal-title":"Nature Medicine"},{"key":"pcbi.1011790.ref021","doi-asserted-by":"crossref","first-page":"106212","DOI":"10.1016\/j.compbiomed.2022.106212","article-title":"D3AI-Spike: A deep learning platform for predicting binding affinity between SARS-CoV-2 spike receptor binding domain with multiple amino acid mutations and human angiotensin-converting enzyme 2","volume":"151","author":"J Han","year":"2022","journal-title":"Computers in Biology and Medicine"},{"issue":"13","key":"pcbi.1011790.ref022","first-page":"30494","article-title":"GISAID: from vision to reality","volume":"22","author":"Y Shu","year":"2017","journal-title":"Eurosurveillance"},{"issue":"D1","key":"pcbi.1011790.ref023","doi-asserted-by":"crossref","first-page":"D158","DOI":"10.1093\/nar\/gkw1099","article-title":"UniProt: the universal protein knowledgebase","volume":"45","year":"2017","journal-title":"Nucleic acids research"},{"key":"pcbi.1011790.ref024","unstructured":"Keskar NS, McCann B, Varshney LR, Xiong C, Socher R. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:190905858. 2019;."},{"key":"pcbi.1011790.ref025","unstructured":"Lu X, Welleck S, Jiang L, Hessel J, Qin L, West P, et al. Quark: Controllable text generation with reinforced unlearning. arXiv preprint arXiv:220513636. 2022;."},{"issue":"1","key":"pcbi.1011790.ref026","first-page":"5485","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"C Raffel","year":"2020","journal-title":"The Journal of Machine Learning Research"},{"issue":"8","key":"pcbi.1011790.ref027","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"A Radford","year":"2019","journal-title":"OpenAI blog"},{"key":"pcbi.1011790.ref028","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need. Advances in neural information processing systems. 2017;30."},{"key":"pcbi.1011790.ref029","unstructured":"Holtzman A, Buys J, Du L, Forbes M, Choi Y. The curious case of neural text degeneration. arXiv preprint arXiv:190409751. 2019;."},{"key":"pcbi.1011790.ref030","doi-asserted-by":"crossref","unstructured":"Heo Y, Manikandan G, Ramachandran A, Chen D. Comprehensive Evaluation of Error-Correction Methodologies for Genome Sequencing Data. Exon Publications. 2021; p. 89\u2013108.","DOI":"10.36255\/exonpublications.bioinformatics.2021.ch6"},{"issue":"1","key":"pcbi.1011790.ref031","first-page":"147","article-title":"A learning algorithm for Boltzmann machines","volume":"9","author":"DH Ackley","year":"1985","journal-title":"Cognitive science"},{"key":"pcbi.1011790.ref032","doi-asserted-by":"crossref","unstructured":"Ficler J, Goldberg Y. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:170702633. 2017;.","DOI":"10.18653\/v1\/W17-4912"},{"key":"pcbi.1011790.ref033","unstructured":"Caccia M, Caccia L, Fedus W, Larochelle H, Pineau J, Charlin L. Language gans falling short. arXiv preprint arXiv:181102549. 2018;."},{"issue":"11","key":"pcbi.1011790.ref034","doi-asserted-by":"crossref","first-page":"1403","DOI":"10.1038\/s41564-020-0770-5","article-title":"A dynamic nomenclature proposal for SARS-CoV-2 lineages to assist genomic epidemiology","volume":"5","author":"A Rambaut","year":"2020","journal-title":"Nature microbiology"},{"key":"pcbi.1011790.ref035","first-page":"e939580","article-title":"The XBB. 1.5 (\u2018Kraken\u2019) Subvariant of Omicron SARS-CoV-2 and its Rapid Global Spread","volume":"29","author":"DV Parums","year":"2023","journal-title":"Medical Science Monitor: International Medical Journal of Experimental and Clinical Research"},{"issue":"5","key":"pcbi.1011790.ref036","doi-asserted-by":"crossref","first-page":"2068883","DOI":"10.1080\/21645515.2022.2068883","article-title":"Delta variant (B. 1.617. 2) of SARS-CoV-2: Mutations, impact, challenges and possible solutions","volume":"18","author":"M Dhawan","year":"2022","journal-title":"Human Vaccines & Immunotherapeutics"},{"issue":"1","key":"pcbi.1011790.ref037","doi-asserted-by":"crossref","first-page":"167","DOI":"10.3390\/v15010167","article-title":"A detailed overview of SARS-CoV-2 omicron: its sub-variants, mutations and pathophysiology, clinical characteristics, immunological landscape, immune escape, and therapies","volume":"15","author":"S Chatterjee","year":"2023","journal-title":"Viruses"},{"issue":"4","key":"pcbi.1011790.ref038","doi-asserted-by":"crossref","first-page":"1329","DOI":"10.1097\/MS9.0000000000000469","article-title":"Emergence BQ. 1 and BQ. 1.1 as newly identified omicron subvariants: current scenario and future outlook\u2013an update","volume":"85","author":"S Akash","year":"2023","journal-title":"Annals of Medicine and Surgery"},{"issue":"2","key":"pcbi.1011790.ref039","doi-asserted-by":"crossref","first-page":"e239","DOI":"10.1002\/mco2.239","article-title":"The rapid rise of SARS-CoV-2 Omicron subvariants with immune evasion properties: XBB. 1.5 and BQ. 1.1 subvariants","volume":"4","author":"D Ao","year":"2023","journal-title":"MedComm"},{"issue":"1","key":"pcbi.1011790.ref040","doi-asserted-by":"crossref","first-page":"2800","DOI":"10.1038\/s41467-023-38435-3","article-title":"Virological characteristics of the SARS-CoV-2 XBB variant derived from recombination of two Omicron subvariants","volume":"14","author":"T Tamura","year":"2023","journal-title":"Nature communications"},{"issue":"6","key":"pcbi.1011790.ref041","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.3390\/v14061239","article-title":"Recombination in Coronaviruses, with a Focus on SARS-CoV-2","volume":"14","author":"D Focosi","year":"2022","journal-title":"Viruses"},{"issue":"7","key":"pcbi.1011790.ref042","doi-asserted-by":"crossref","first-page":"1114","DOI":"10.1038\/s41587-021-01146-5","article-title":"Learning protein fitness models from evolutionary and assay-labeled data","volume":"40","author":"C Hsu","year":"2022","journal-title":"Nature biotechnology"},{"key":"pcbi.1011790.ref043","unstructured":"Narang S, Chowdhery A. Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance; 2022."},{"key":"pcbi.1011790.ref044","unstructured":"PyTorch T. PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever; 2023."},{"key":"pcbi.1011790.ref045","unstructured":"Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, et al. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:210609685. 2021;."},{"key":"pcbi.1011790.ref046","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s10020-018-0024-7","article-title":"Advances in therapeutic use of a drug-stimulated translational readthrough of premature termination codons","volume":"24","author":"M Dabrowski","year":"2018","journal-title":"Molecular medicine"},{"key":"pcbi.1011790.ref047","doi-asserted-by":"crossref","unstructured":"Kindratenko V, Mu D, Zhan Y, Maloney J, Hashemi SH, Rabe B, et al. Hal: Computer system for scalable deep learning. In: Practice and experience in advanced research computing; 2020. p. 41\u201348.","DOI":"10.1145\/3311790.3396649"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011790","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T00:00:00Z","timestamp":1706659200000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011790","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T13:41:13Z","timestamp":1706708473000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011790"}},"subtitle":[],"editor":[{"given":"Samuel V.","family":"Scarpino","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,1,19]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1,19]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011790","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.05.10.540124","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,19]]}}}