{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T07:49:31Z","timestamp":1775980171505,"version":"3.50.1"},"posted":{"date-parts":[[2025,6,1]]},"group-title":"Bioinformatics","reference-count":45,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,6,1]],"date-time":"2025-06-01T00:00:00Z","timestamp":1748736000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"INBRE","award":["P20GM103446"],"award-info":[{"award-number":["P20GM103446"]}]},{"name":"INBRE","award":["NIH S10OD028725"],"award-info":[{"award-number":["NIH S10OD028725"]}]},{"name":"NSF","award":["1736030"],"award-info":[{"award-number":["1736030"]}]},{"name":"NSF","award":["2025567"],"award-info":[{"award-number":["2025567"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,6,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>The transformer architecture in deep learning has revolutionized protein sequence analysis. Recent advancements in protein language models have paved the way for significant progress across various domains, including protein function and structure prediction, multiple sequence alignments and mutation effect prediction. A protein language model is commonly trained on individual proteins, ignoring the interdependencies between sequences within a genome. However, biological understanding reveals that protein\u2013protein interactions span entire genomic regions, underscoring the limitations of focusing solely on individual proteins. To address these limitations, we propose a novel approach that extends the context size of transformer models across the entire viral genome. By training on large genomic fragments, our method captures long-range interprotein interactions and encodes protein sequences with integrated information from distant proteins within the same genome, offering substantial benefits in various tasks. Viruses, with their densely packed genomes, minimal intergenic regions, and protein annotation challenges, are ideal candidates for genome-wide learning. We introduce a long-context protein language model, trained on entire viral genomes, leveraging a sparse attention mechanism based on protein\u2013protein interactions. Our semi-supervised approach supports long sequences of up to 61,000 amino acids (aa). Our evaluations demonstrate that the resulting embeddings significantly surpass those generated by single-protein models and outperform alternative large-context architectures that rely on static masking or non-transformer frameworks.<\/jats:p>","DOI":"10.1101\/2025.05.29.656907","type":"posted-content","created":{"date-parts":[[2025,6,1]],"date-time":"2025-06-01T22:35:11Z","timestamp":1748817311000},"source":"Crossref","is-referenced-by-count":1,"title":["Extending Protein Language Models to a Viral Genomic Scale Using Biologically Induced Sparse Attention"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-2394-6283","authenticated-orcid":false,"given":"Thibaut","family":"Dejean","sequence":"first","affiliation":[{"name":"Department of Information and Computer Sciences, University of Hawaii, Honolulu, HI"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Barbra D.","family":"Ferrell","sequence":"additional","affiliation":[{"name":"Department of Computer & Information Sciences, Delaware Biotechnology Institute, University of Delaware, Newark, DE"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2935-6863","authenticated-orcid":false,"given":"William","family":"Harrigan","sequence":"additional","affiliation":[{"name":"Hawai\u2019i Institute of Marine Biology, University of Hawai\u2019i, Kaneohe, HI"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zachary D.","family":"Schreiber","sequence":"additional","affiliation":[{"name":"Department of Computer & Information Sciences, Delaware Biotechnology Institute, University of Delaware, Newark, DE"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0060-759X","authenticated-orcid":false,"given":"Rajan","family":"Sawhney","sequence":"additional","affiliation":[{"name":"Department of Information and Computer Sciences, University of Hawaii, Honolulu, HI"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5797-7128","authenticated-orcid":false,"given":"K. Eric","family":"Wommack","sequence":"additional","affiliation":[{"name":"Department of Plant & Soil Sciences, Delaware Biotechnology Institute, University of Delaware, Newark, DE"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3398-6932","authenticated-orcid":false,"given":"Shawn W.","family":"Polson","sequence":"additional","affiliation":[{"name":"Department of Computer & Information Sciences, Delaware Biotechnology Institute, University of Delaware, Newark, DE"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5185-0643","authenticated-orcid":false,"given":"Mahdi","family":"Belcaid","sequence":"additional","affiliation":[{"name":"Department of Information and Computer Sciences, University of Hawaii, Honolulu, HI"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2026041200264454000_2025.05.29.656907v2.1","year":"2017","journal-title":"Deep Recurrent Neural Network for Protein Function Prediction from Sequence"},{"key":"2026041200264454000_2025.05.29.656907v2.2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.82819"},{"key":"2026041200264454000_2025.05.29.656907v2.3","unstructured":"Zeming Lin , Halil Akin , Roshan Rao , Brian Hie , Zhongkai Zhu , Wenting Lu , Allan dos Santos Costa , Maryam Fazel-Zarandi , Tom Sercu , Sal Candido , and Alexander Rives . Language models of protein sequences at the scale of evolution enable accurate structure prediction. July 2022. Pages: 2022.07.20.500902 Section: New Results."},{"key":"2026041200264454000_2025.05.29.656907v2.4","year":"2021","journal-title":"ProtTrans: Towards Cracking the Language of Life\u2019s Code Through Self-Supervised Deep Learning and High Performance Computing"},{"key":"2026041200264454000_2025.05.29.656907v2.5","year":"2022","journal-title":"ProGen2: Exploring the Boundaries of Protein Language Models"},{"key":"2026041200264454000_2025.05.29.656907v2.6","doi-asserted-by":"crossref","unstructured":"Claire D. McWhite and Mona Singh . Vector-clustering Multiple Sequence Alignment: Aligning into the twilight zone of protein sequence similarity with protein language models, October 2022.","DOI":"10.1101\/2022.10.21.513099"},{"key":"2026041200264454000_2025.05.29.656907v2.7","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-024-05699-5"},{"key":"2026041200264454000_2025.05.29.656907v2.8","unstructured":"Zeming Lin , Halil Akin , Roshan Rao , Brian Hie , Zhongkai Zhu , Wenting Lu , Nikita Smetanin , Robert Verkuil , Ori Kabeli , Yaniv Shmueli , Allan dos Santos Costa , Maryam Fazel-Zarandi , Tom Sercu , Salvatore Candido , and Alexander Rives . Evolutionary-scale prediction of atomic level protein structure with a language model, October 2022. Pages: 2022.07.20.500902 Section: New Results."},{"key":"2026041200264454000_2025.05.29.656907v2.9","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0149166"},{"key":"2026041200264454000_2025.05.29.656907v2.10","doi-asserted-by":"publisher","DOI":"10.1101\/gr.277567.122"},{"key":"2026041200264454000_2025.05.29.656907v2.11","doi-asserted-by":"publisher","DOI":"10.1186\/s13578-021-00644-y"},{"key":"2026041200264454000_2025.05.29.656907v2.12","year":"2024","journal-title":"Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models"},{"key":"2026041200264454000_2025.05.29.656907v2.13","year":"2023","journal-title":"Attention Is All You Need"},{"key":"2026041200264454000_2025.05.29.656907v2.14","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gku1207"},{"key":"2026041200264454000_2025.05.29.656907v2.15","year":"2023","journal-title":"RoFormer: Enhanced Transformer with Rotary Position Embedding"},{"key":"2026041200264454000_2025.05.29.656907v2.16","year":"2022","journal-title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation"},{"key":"2026041200264454000_2025.05.29.656907v2.17","year":"2023","journal-title":"Extending Context Window of Large Language Models via Positional Interpolation"},{"key":"2026041200264454000_2025.05.29.656907v2.18","year":"2023","journal-title":"YaRN: Efficient Context Window Extension of Large Language Models"},{"key":"2026041200264454000_2025.05.29.656907v2.19","year":"2022","journal-title":"FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness"},{"key":"2026041200264454000_2025.05.29.656907v2.20","doi-asserted-by":"crossref","unstructured":"Philippe Tillet , H. T. Kung , and David Cox . Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10\u201319, Phoenix AZ USA, June 2019. ACM.","DOI":"10.1145\/3315508.3329973"},{"key":"2026041200264454000_2025.05.29.656907v2.21","unstructured":"Houda Bouamor , Juan Pino , and Kalika Bali , editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Singapore, December 2023."},{"key":"2026041200264454000_2025.05.29.656907v2.22","year":"2023","journal-title":"LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models"},{"key":"2026041200264454000_2025.05.29.656907v2.23","doi-asserted-by":"crossref","unstructured":"Genta Indra Winata , Samuel Cahyawijaya , Zhaojiang Lin , Zihan Liu , and Pascale Fung . Lightweight and efficient end-to-end speech recognition using low-rank transformer, 2020.","DOI":"10.1109\/ICASSP40776.2020.9053878"},{"key":"2026041200264454000_2025.05.29.656907v2.24","year":"2024","journal-title":"SimA: Simple Softmax-free Attention for Vision Transformers"},{"key":"2026041200264454000_2025.05.29.656907v2.25","year":"2020","journal-title":"Longformer: The Long-Document Transformer"},{"key":"2026041200264454000_2025.05.29.656907v2.26","year":"2024","journal-title":"LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models"},{"key":"2026041200264454000_2025.05.29.656907v2.27","year":"2021","journal-title":"Big Bird: Transformers for Longer Sequences"},{"key":"2026041200264454000_2025.05.29.656907v2.28","year":"2020","journal-title":"Efficient Content-Based Sparse Attention with Routing Transformers"},{"key":"2026041200264454000_2025.05.29.656907v2.29","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btac020"},{"key":"2026041200264454000_2025.05.29.656907v2.30","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btu739"},{"key":"2026041200264454000_2025.05.29.656907v2.31","doi-asserted-by":"crossref","unstructured":"Cody Martin , Anthony Gitter , and Karthik Anantharaman . Protein Set Transformer: A protein-based genome language model to power high diversity viromics, July 2024. Pages: 2024.07.26.605391 Section: New Results.","DOI":"10.1101\/2024.07.26.605391"},{"key":"2026041200264454000_2025.05.29.656907v2.32","doi-asserted-by":"crossref","unstructured":"Yunha Hwang , Andre Cornman , Elizabeth Kellogg , Sergey Ovchinnikov , and Peter Girguis . Genomic language model predicts protein co-regulation and function. 2024.","DOI":"10.1101\/2023.04.07.536042"},{"key":"2026041200264454000_2025.05.29.656907v2.33","doi-asserted-by":"crossref","unstructured":"Eric Nguyen , Michael Poli , Matthew G. Durrant , Armin W. Thomas , Brian Kang , Jeremy Sullivan , Madelena Y. Ng , Ashley Lewis , Aman Patel , Aaron Lou , Stefano Ermon , Stephen A. Baccus , Tina Hernandez-Boussard , Christopher R\u00e9 , Patrick D. Hsu , and Brian L. Hie . Sequence modeling and design from molecular to genome scale with Evo, February 2024. Pages: 2024.02.27.582234 Section: New Results.","DOI":"10.1126\/science.ado9336"},{"key":"2026041200264454000_2025.05.29.656907v2.34","year":"2024","journal-title":"DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome"},{"key":"2026041200264454000_2025.05.29.656907v2.35","unstructured":"Maxim Zvyagin , Alexander Brace , Kyle Hippe , Yuntian Deng , Bin Zhang , Cindy Orozco Bohorquez , Austin Clyde , Bharat Kale , Danilo Perez-Rivera , Heng Ma , Carla M. Mann , Michael Irvin , J. Gregory Pauloski , Logan Ward , Valerie Hayot , Murali Emani , Sam Foreman , Zhen Xie , Diangen Lin , Maulik Shukla , Weili Nie , Josh Romero , Christian Dallago , Arash Vahdat , Chaowei Xiao , Thomas Gibbs , Ian Foster , James J. Davis , Michael E. Papka , Thomas Brettin , Rick Stevens , Anima Anandkumar , Venkatram Vishwanath , and Arvind Ramanathan . GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics, October 2022. Pages: 2022.10.10.511571 Section: New Results."},{"key":"2026041200264454000_2025.05.29.656907v2.36","year":"2023","journal-title":"Hyena Hierarchy: Towards Larger Convolutional Language Models"},{"key":"2026041200264454000_2025.05.29.656907v2.37","unstructured":"Michael Poli , Jue Wang , Stefano Massaroli , Jeffrey Quesnelle , Ryan Carlow , Eric Nguyen , and Armin Thomas . Stripedhyena: Moving beyond transformers with hybrid signal processing models. 12 2023."},{"key":"2026041200264454000_2025.05.29.656907v2.38","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2016239118"},{"key":"2026041200264454000_2025.05.29.656907v2.39","unstructured":"Roshan Rao , Joshua Meier , Tom Sercu , Sergey Ovchinnikov , and Alexander Rives . Transformer protein language models are unsupervised structure learners. 12 2020."},{"key":"2026041200264454000_2025.05.29.656907v2.40","unstructured":"torch.sparse_bsr_tensor \u2014 PyTorch 2.6 documentation."},{"key":"2026041200264454000_2025.05.29.656907v2.41","doi-asserted-by":"crossref","unstructured":"Rachel A. Keown , Jacob T. Dums , Phillip J. Brumm , Joyanne MacDonald , David A. Mead , Barbra D. Ferrell , Ryan M. Moore , Amelia O. Harrison , Shawn W. Polson , and K. Eric Wommack . Novel viral dna polymerases from metagenomes suggest genomic sources of strand-displacing biochemical phenotypes. Frontiers in Microbiology, Volume 13 \u2013 2022, 2022.","DOI":"10.3389\/fmicb.2022.858366"},{"key":"2026041200264454000_2025.05.29.656907v2.42","doi-asserted-by":"publisher","DOI":"10.1016\/j.chom.2022.06.008"},{"key":"2026041200264454000_2025.05.29.656907v2.43","doi-asserted-by":"publisher","DOI":"10.3389\/fmicb.2018.03053"},{"key":"2026041200264454000_2025.05.29.656907v2.44","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bty1057"},{"key":"2026041200264454000_2025.05.29.656907v2.45","doi-asserted-by":"publisher","DOI":"10.1093\/NAR\/GKAC1000"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.05.29.656907","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T07:26:54Z","timestamp":1775978814000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.1101\/2025.05.29.656907"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,1]]},"references-count":45,"URL":"https:\/\/doi.org\/10.1101\/2025.05.29.656907","relation":{},"subject":[],"published":{"date-parts":[[2025,6,1]]},"subtype":"preprint"}}