{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T07:27:39Z","timestamp":1784359659514,"version":"3.55.0"},"reference-count":69,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T00:00:00Z","timestamp":1698364800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"Exascale Computing Project"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:p>We seek to transform how new and emergent variants of pandemic-causing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pre-training on over 110 million prokaryotic gene sequences and fine-tuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole-genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.<\/jats:p>","DOI":"10.1177\/10943420231201154","type":"journal-article","created":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T13:08:56Z","timestamp":1698412136000},"page":"683-705","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":81,"title":["GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics"],"prefix":"10.1177","volume":"37","author":[{"given":"Maxim","family":"Zvyagin","sequence":"first","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Brace","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle","family":"Hippe","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuntian","family":"Deng","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"},{"name":"Harvard University, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Cerebras Inc., San Jose, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cindy Orozco","family":"Bohorquez","sequence":"additional","affiliation":[{"name":"Cerebras Inc., San Jose, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Austin","family":"Clyde","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bharat","family":"Kale","sequence":"additional","affiliation":[{"name":"Computer Science Department, Northern Illinois University, DeKalb, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danilo","family":"Perez-Rivera","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"New York University, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heng","family":"Ma","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carla M.","family":"Mann","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Irvin","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7316-3922","authenticated-orcid":false,"given":"Defne G.","family":"Ozgulbas","sequence":"additional","affiliation":[{"name":"Department of Biochemistry, University of Illinois-Urbana Champaign, Champaign, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalia","family":"Vassilieva","sequence":"additional","affiliation":[{"name":"Cerebras Inc., San Jose, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6547-6902","authenticated-orcid":false,"given":"James Gregory","family":"Pauloski","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Logan","family":"Ward","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valerie","family":"Hayot-Sasson","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"},{"name":"Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Murali","family":"Emani","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sam","family":"Foreman","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhen","family":"Xie","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6778-8563","authenticated-orcid":false,"given":"Diangen","family":"Lin","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maulik","family":"Shukla","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weili","family":"Nie","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josh","family":"Romero","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Dallago","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"},{"name":"Computer Science Department, Technical University of Munich, Munich,Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arash","family":"Vahdat","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chaowei","family":"Xiao","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"},{"name":"Department of Biochemistry, University of Illinois-Urbana Champaign, Champaign, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Gibbs","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2129-5269","authenticated-orcid":false,"given":"Ian","family":"Foster","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James J.","family":"Davis","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael E.","family":"Papka","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Computer Science Department, University of Illinois Chicago, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Brettin","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Computing, Environment and Life Sciences Directorate, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rick","family":"Stevens","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Department of Computer Science, University of Chicago, Hyde Park, IL, USA"},{"name":"Computing, Environment and Life Sciences Directorate, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anima","family":"Anandkumar","sequence":"additional","affiliation":[{"name":"NVIDIA Inc., Santa Clara, CA, USA"},{"name":"Computer Science Department, California Institute of Technology, Pasadena, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Venkatram","family":"Vishwanath","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"},{"name":"Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1622-5488","authenticated-orcid":false,"given":"Arvind","family":"Ramanathan","sequence":"additional","affiliation":[{"name":"Data Science and Learning Division, Argonne National Laboratory, Lemont, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2023,10,27]]},"reference":[{"key":"bibr1-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-021-01252-x"},{"key":"bibr2-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1145\/3307681.3325400"},{"key":"bibr3-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1009624"},{"key":"bibr4-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/HiPC.2018.00014"},{"key":"bibr5-10943420231201154","volume-title":"Radical-cybertools: Middleware Building Blocks for Scalable Science","author":"Balasubramanian V","year":"2019"},{"key":"bibr6-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2023.106618"},{"key":"bibr7-10943420231201154","volume-title":"Gpu Performance Analysis and Optimisation","author":"Bradley T","year":"2012"},{"key":"bibr8-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1126\/science.abc5902"},{"key":"bibr9-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1016\/j.cytogfr.2021.06.001"},{"key":"bibr10-10943420231201154","doi-asserted-by":"publisher","DOI":"10.3389\/fmicb.2016.00118"},{"key":"bibr11-10943420231201154","volume-title":"Flops Profiler - Deepspeed","author":"Deepspeed","year":"2022"},{"key":"bibr12-10943420231201154","volume-title":"Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding","author":"Devlin J","year":"2018"},{"key":"bibr13-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-03665-3"},{"key":"bibr14-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1088\/2515-7639\/ab0c3d"},{"key":"bibr15-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3095381"},{"key":"bibr16-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1101\/2022.08.31.505981"},{"key":"bibr17-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1016\/j.chom.2020.11.007"},{"key":"bibr18-10943420231201154","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2005.08100"},{"key":"bibr19-10943420231201154","volume-title":"Training Giant Neural Networks Using Weight Streaming on Cerebras Wafer-Scale Systems","author":"Hall S","year":"2021"},{"key":"bibr20-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2022.3152247"},{"key":"bibr21-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1126\/science.abd7331"},{"key":"bibr22-10943420231201154","first-page":"6840","volume":"33","author":"Ho J","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr23-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/tpds.2021.3082815."},{"key":"bibr24-10943420231201154","volume-title":"Transformers: State-Of-The-Art Machine Learning For Pytorch, Tensorflow, And Jax","author":"Hugging face","year":"2022"},{"key":"bibr25-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00300"},{"key":"bibr26-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2380-z"},{"key":"bibr27-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-03819-2"},{"key":"bibr28-10943420231201154","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2001.08361"},{"key":"bibr29-10943420231201154","first-page":"4171","volume-title":"Proceedings of NAACL-HLT","author":"Kenton JDMWC","year":"2019"},{"key":"bibr30-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz305"},{"key":"bibr31-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1101\/2022.07.20.500902"},{"key":"bibr32-10943420231201154","volume-title":"Lost in the Middle: How Language Models Use Long Contexts","author":"Liu NF","year":"2023"},{"key":"bibr33-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.abk3445"},{"key":"bibr34-10943420231201154","unstructured":"Moritz P, Nishihara R, Wang S, et al. (2018) Ray: a distributed framework for emerging AI applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). Carlsbad, CA, 2018, pp. 561\u2013577. https:\/\/www.usenix.org\/conference\/osdi18\/presentation\/moritz"},{"key":"bibr35-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1016\/0022-2836(70)90057-4"},{"key":"bibr36-10943420231201154","volume-title":"Automated Mixed Precision","author":"Nvidia","year":"2022"},{"key":"bibr37-10943420231201154","volume-title":"Nvidia Nsight Systems","author":"NVIDIA","year":"2022"},{"key":"bibr38-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2021.06.049"},{"key":"bibr39-10943420231201154","first-page":"436","volume":"5","author":"Page AJ","year":"2016","journal-title":"Microbial Genomics"},{"key":"bibr40-10943420231201154","volume-title":"Towards Coherent and Consistent Use of Entities in Narrative Generation","author":"Papalampidi P","year":"2022"},{"key":"bibr41-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1145\/3581784.3607047"},{"key":"bibr42-10943420231201154","volume-title":"Pytorch Lightning","author":"Pytorch","year":"2022"},{"key":"bibr43-10943420231201154","volume-title":"Improving Language Understanding by Generative Pre-training","author":"Radford A","year":"2018"},{"key":"bibr44-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"bibr45-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41564-020-0770-5"},{"key":"bibr46-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3019989"},{"key":"bibr47-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3406703"},{"key":"bibr48-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2016239118"},{"key":"bibr49-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/XLOOP49562.2019.00010"},{"key":"bibr50-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1515387112"},{"key":"bibr51-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btu033"},{"key":"bibr52-10943420231201154","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.62"},{"key":"bibr53-10943420231201154","volume-title":"Chapterbreak: A Challenge Dataset for Long-Range Language Models","author":"Sun S","year":"2022"},{"key":"bibr54-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00459-8"},{"key":"bibr55-10943420231201154","volume-title":"June 2022 \u2013 Top500","author":"Top500","year":"2022"},{"key":"bibr56-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41588-021-00862-7"},{"key":"bibr57-10943420231201154","volume-title":"Well-read Students Learn Better: On the Importance of Pre-training Compact Models","author":"Turc I","year":"2019"},{"key":"bibr58-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-022-00457-9"},{"key":"bibr59-10943420231201154","first-page":"397","volume":"30","author":"Vaswani A","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr60-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00142"},{"key":"bibr61-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1101\/2022.08.08.22278553"},{"key":"bibr62-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gks528"},{"key":"bibr63-10943420231201154","unstructured":"Wang RE, Durmus E, Goodman N, et al. (2022) Language modeling via stochastic processes. In International Conference on Learning Representations. 2022"},{"key":"bibr64-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/MLHPC54614.2021.00007"},{"key":"bibr65-10943420231201154","volume-title":"Emergent Abilities of Large Language Models","author":"Wei J","year":"2022"},{"key":"bibr66-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2013.99"},{"key":"bibr67-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btw412"},{"key":"bibr68-10943420231201154","volume-title":"Opt: Open Pre-trained Transformer Language Models","author":"Zhang S","year":"2022"},{"key":"bibr69-10943420231201154","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0998-x"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420231201154","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10943420231201154","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10943420231201154","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:17:32Z","timestamp":1777450652000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10943420231201154"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,27]]},"references-count":69,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["10.1177\/10943420231201154"],"URL":"https:\/\/doi.org\/10.1177\/10943420231201154","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.10.10.511571","asserted-by":"object"}]},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,27]]}}}