{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T17:35:11Z","timestamp":1793122511555,"version":"4.5.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012646","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T00:00:00Z","timestamp":1734480000000}}],"reference-count":52,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2024,12,6]],"date-time":"2024-12-06T00:00:00Z","timestamp":1733443200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Antibodies are proteins produced by the immune system that can identify and neutralise a wide variety of antigens with high specificity and affinity, and constitute the most successful class of biotherapeutics. With the advent of next-generation sequencing, billions of antibody sequences have been collected in recent years, though their application in the design of better therapeutics has been constrained by the sheer volume and complexity of the data. To address this challenge, we present IgBert and IgT5, the best performing antibody-specific language models developed to date which can consistently handle both paired and unpaired variable region sequences as input. These models are trained comprehensively using the more than two billion unpaired sequences and two million paired sequences of light and heavy chains present in the Observed Antibody Space dataset. We show that our models outperform existing antibody and protein language models on a diverse range of design and regression tasks relevant to antibody engineering. This advancement marks a significant leap forward in leveraging machine learning, large scale data sets and high-performance computing for enhancing antibody design for therapeutic development.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012646","type":"journal-article","created":{"date-parts":[[2024,12,6]],"date-time":"2024-12-06T13:37:09Z","timestamp":1733492229000},"page":"e1012646","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":73,"title":["Large scale paired antibody language models"],"prefix":"10.1371","volume":"20","author":[{"given":"Henry","family":"Kenlay","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0445-7863","authenticated-orcid":true,"given":"Fr\u00e9d\u00e9ric A.","family":"Dreyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aleksandr","family":"Kovaltsuk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dom","family":"Miketa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3004-2119","authenticated-orcid":true,"given":"Douglas","family":"Pires","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charlotte M.","family":"Deane","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,12,6]]},"reference":[{"key":"pcbi.1012646.ref001","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1016\/j.nbt.2011.03.012","article-title":"Analysis of the antibody structure based on high-resolution crystallographic studies","volume":"28","author":"JE Narciso","year":"2011","journal-title":"New biotechnology"},{"key":"pcbi.1012646.ref002","doi-asserted-by":"crossref","DOI":"10.1002\/pro.2874","article-title":"The diversity of H3 loops determines the antigen-binding tendencies of antibody CDR loops","volume":"25","author":"Y Tsuchiya","year":"2016","journal-title":"Protein science: a publication of the Protein Society"},{"key":"pcbi.1012646.ref003","doi-asserted-by":"crossref","DOI":"10.1002\/prot.25291","article-title":"The H3 loop of antibodies shows unique structural characteristics: CDR H3 of Antibodies Shows Unique Structural Characteristics","volume":"85","author":"C Regep","year":"2017","journal-title":"Proteins: Structure, Function, and Bioinformatics"},{"key":"pcbi.1012646.ref004","doi-asserted-by":"crossref","DOI":"10.3389\/fimmu.2018.00462","article-title":"Analyzing Immunoglobulin Repertoires","volume":"9","author":"N Chaudhary","year":"2018","journal-title":"Frontiers in Immunology"},{"issue":"1","key":"pcbi.1012646.ref005","doi-asserted-by":"crossref","first-page":"vbac046","DOI":"10.1093\/bioadv\/vbac046","article-title":"AbLang: an antibody language model for completing antibody sequences","volume":"2","author":"TH Olsen","year":"2022","journal-title":"Bioinformatics Advances"},{"key":"pcbi.1012646.ref006","article-title":"Generative language modeling for antibody design","author":"RW Shuai","year":"2022","journal-title":"bioRxiv"},{"key":"pcbi.1012646.ref007","unstructured":"Gruver N, Stanton S, Frey NC, Rudner TGJ, Hotzel I, Lafrance-Vanasse J, et al. Protein Design with Guided Discrete Diffusion; 2023."},{"issue":"1","key":"pcbi.1012646.ref008","doi-asserted-by":"crossref","first-page":"2389","DOI":"10.1038\/s41467-023-38063-x","article-title":"Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies","volume":"14","author":"JA Ruffolo","year":"2023","journal-title":"Nature Communications"},{"key":"pcbi.1012646.ref009","unstructured":"H\u00f8ie M, Hummer A, Olsen T, Nielsen M, Deane C. AntiFold: Improved antibody structure design using inverse folding. In: NeurIPS 2023 Generative AI and Biology (GenBio) Workshop; 2023. Available from: https:\/\/openreview.net\/forum?id=bxZMKHtlL6."},{"issue":"7","key":"pcbi.1012646.ref010","doi-asserted-by":"crossref","first-page":"100513","DOI":"10.1016\/j.patter.2022.100513","article-title":"Deciphering the language of antibodies using self-supervised learning","volume":"3","author":"J Leem","year":"2022","journal-title":"Patterns"},{"issue":"1","key":"pcbi.1012646.ref011","doi-asserted-by":"crossref","first-page":"2020203","DOI":"10.1080\/19420862.2021.2020203","article-title":"BioPhi: A platform for antibody design, humanization, and humanness evaluation based on natural antibody repertoires and deep learning","volume":"14","author":"D Prihoda","year":"2022","journal-title":"mAbs"},{"issue":"1","key":"pcbi.1012646.ref012","doi-asserted-by":"crossref","first-page":"2163584","DOI":"10.1080\/19420862.2022.2163584","article-title":"Toward generalizable prediction of antibody thermostability using machine learning on sequence and structure features","volume":"15","author":"A Harmalkar","year":"2023","journal-title":"mAbs"},{"issue":"8","key":"pcbi.1012646.ref013","doi-asserted-by":"crossref","first-page":"2502","DOI":"10.4049\/jimmunol.1800708","article-title":"Observed Antibody Space: A Resource for Data Mining Next-Generation Sequencing of Antibody Repertoires","volume":"201","author":"A Kovaltsuk","year":"2018","journal-title":"The Journal of Immunology"},{"issue":"1","key":"pcbi.1012646.ref014","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1002\/pro.4205","article-title":"Observed Antibody Space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences","volume":"31","author":"TH Olsen","year":"2022","journal-title":"Protein Science"},{"key":"pcbi.1012646.ref015","article-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","author":"J Devlin","year":"2018","journal-title":"CoRR"},{"issue":"1","key":"pcbi.1012646.ref016","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"C Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"key":"pcbi.1012646.ref017","doi-asserted-by":"crossref","unstructured":"Dreyer FA, Kenlay H. Large scale paired antibody language models; 2024. Available from: https:\/\/zenodo.org\/doi\/10.5281\/zenodo.10876908.","DOI":"10.1371\/journal.pcbi.1012646"},{"issue":"15","key":"pcbi.1012646.ref018","doi-asserted-by":"crossref","first-page":"e2016239118","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"A Rives","year":"2021","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"6637","key":"pcbi.1012646.ref019","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Z Lin","year":"2023","journal-title":"Science"},{"issue":"10","key":"pcbi.1012646.ref020","doi-asserted-by":"crossref","first-page":"7112","DOI":"10.1109\/TPAMI.2021.3095381","article-title":"ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning","volume":"44","author":"A Elnaggar","year":"2022","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"8","key":"pcbi.1012646.ref021","doi-asserted-by":"crossref","first-page":"1099","DOI":"10.1038\/s41587-022-01618-2","article-title":"Large language models generate functional protein sequences across diverse families","volume":"41","author":"A Madani","year":"2023","journal-title":"Nature Biotechnology"},{"issue":"D1","key":"pcbi.1012646.ref022","doi-asserted-by":"crossref","first-page":"D523","DOI":"10.1093\/nar\/gkac1052","article-title":"UniProt: the Universal Protein Knowledgebase in 2023","volume":"51","author":"TU Consortium","year":"2022","journal-title":"Nucleic Acids Research"},{"issue":"1","key":"pcbi.1012646.ref023","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1093\/nar\/28.1.45","article-title":"The SWISS-PROT protein sequence database and its supplement TrEMBL in 2000","volume":"28","author":"A Bairoch","year":"2000","journal-title":"Nucleic Acids Research"},{"issue":"4","key":"pcbi.1012646.ref024","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1016\/j.cels.2022.01.003","article-title":"Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins","volume":"13","author":"BL Hie","year":"2022","journal-title":"Cell Systems"},{"key":"pcbi.1012646.ref025","unstructured":"Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach; 2019."},{"key":"pcbi.1012646.ref026","unstructured":"Ruffolo JA, Gray JJ, Sulam J. Deciphering antibody affinity maturation with language models and weakly supervised learning; 2021."},{"key":"pcbi.1012646.ref027","doi-asserted-by":"crossref","unstructured":"Burbach SM, Briney B. Improving antibody language models with native pairing; 2023.","DOI":"10.1016\/j.patter.2024.100967"},{"key":"pcbi.1012646.ref028","article-title":"Addressing the antibody germline bias and its effect on language models for improved antibody design","author":"TH Olsen","year":"2024","journal-title":"bioRxiv"},{"key":"pcbi.1012646.ref029","article-title":"AbBERT: Learning Antibody Humanness via Masked Language Modeling","author":"D Vashchenko","year":"2022","journal-title":"bioRxiv"},{"key":"pcbi.1012646.ref030","unstructured":"Chu SKS, Wei KY. Generative Antibody Design for Complementary Chain Pairing Sequences through Encoder-Decoder Language Model; 2023."},{"key":"pcbi.1012646.ref031","article-title":"xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein","author":"B Chen","year":"2023","journal-title":"bioRxiv"},{"key":"pcbi.1012646.ref032","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention Is All You Need; 2017."},{"issue":"6","key":"pcbi.1012646.ref033","first-page":"926","article-title":"UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches","volume":"31","author":"the UniProt Consortium","year":"2014","journal-title":"Bioinformatics"},{"issue":"7","key":"pcbi.1012646.ref034","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1038\/s41592-019-0437-4","article-title":"Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold","volume":"16","author":"M Steinegger","year":"2019","journal-title":"Nature Methods"},{"issue":"1","key":"pcbi.1012646.ref035","doi-asserted-by":"crossref","first-page":"2542","DOI":"10.1038\/s41467-018-04964-5","article-title":"Clustering huge protein sequence sets in linear time","volume":"9","author":"M Steinegger","year":"2018","journal-title":"Nature Communications"},{"issue":"11","key":"pcbi.1012646.ref036","doi-asserted-by":"crossref","first-page":"1026","DOI":"10.1038\/nbt.3988","article-title":"MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets","volume":"35","author":"M Steinegger","year":"2017","journal-title":"Nature Biotechnology"},{"key":"pcbi.1012646.ref037","doi-asserted-by":"crossref","unstructured":"Rajbhandari S, Rasley J, Ruwase O, He Y. Zero: Memory optimizations toward training trillion parameter models. In: SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE; 2020. p. 1\u201316.","DOI":"10.1109\/SC41405.2020.00024"},{"key":"pcbi.1012646.ref038","unstructured":"Loshchilov I, Hutter F. Decoupled Weight Decay Regularization. In: International Conference on Learning Representations; 2019. Available from: https:\/\/openreview.net\/forum?id=Bkg6RiCqY7."},{"issue":"13","key":"pcbi.1012646.ref039","doi-asserted-by":"crossref","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"J Kirkpatrick","year":"2017","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1012646.ref040","doi-asserted-by":"crossref","unstructured":"Chen S, Hou Y, Cui Y, Che W, Liu T, Yu X. Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting; 2020.","DOI":"10.18653\/v1\/2020.emnlp-main.634"},{"key":"pcbi.1012646.ref041","unstructured":"Chungyoun M, Ruffolo J, Gray JJ. FLAb: Benchmarking tasks in fitness landscape inference for antibodies. NeurIPS Machine Learning in Structural Biology Workshop. 2023;."},{"key":"pcbi.1012646.ref042","article-title":"Unlocking de novo antibody design with generative artificial intelligence","author":"A Shanehsazzadeh","year":"2023","journal-title":"bioRxiv"},{"issue":"8","key":"pcbi.1012646.ref043","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.1007207","article-title":"Optimizing antibody affinity and stability by the automated design of the variable light-heavy chain interfaces","volume":"15","author":"S Warszawski","year":"2019","journal-title":"PLOS Computational Biology"},{"issue":"4","key":"pcbi.1012646.ref044","doi-asserted-by":"crossref","first-page":"E486","DOI":"10.1073\/pnas.1613231114","article-title":"Mutational landscape of antibody variable domains reveals a switch modulating the interdomain conformational dynamics and antigen binding","volume":"114","author":"P Koenig","year":"2017","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1012646.ref045","doi-asserted-by":"crossref","unstructured":"Gao T, Yao X, Chen D. SimCSE: Simple Contrastive Learning of Sentence Embeddings; 2022.","DOI":"10.18653\/v1\/2021.emnlp-main.552"},{"key":"pcbi.1012646.ref046","article-title":"Learning sequence, structure, and function representations of proteins with language models","author":"T Hamamsy","year":"2023","journal-title":"bioRxiv"},{"key":"pcbi.1012646.ref047","doi-asserted-by":"crossref","unstructured":"Barton J, Galson J, Leem J. Enhancing Antibody Language Models with Structural Information. NeurIPS Machine Learning in Structural Biology workshop. 2023;.","DOI":"10.1101\/2023.12.12.569610"},{"key":"pcbi.1012646.ref048","article-title":"LoRA: Low-Rank Adaptation of Large Language Models","author":"EJ Hu","year":"2021","journal-title":"CoRR"},{"key":"pcbi.1012646.ref049","article-title":"Overcoming catastrophic forgetting in neural networks","author":"J Kirkpatrick","year":"2016","journal-title":"CoRR"},{"key":"pcbi.1012646.ref050","unstructured":"Meier J, Rao R, Verkuil R, Liu J, Sercu T, Rives A. Language models enable zero-shot prediction of the effects of mutations on protein function. In: Ranzato M, Beygelzimer A, Dauphin Y, Liang PS, Vaughan JW, editors. Advances in Neural Information Processing Systems. vol. 34. Curran Associates, Inc.; 2021. p. 29287\u201329303. Available from: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2021\/file\/f51338d736f95dd42427296047067694-Paper.pdf."},{"key":"pcbi.1012646.ref051","article-title":"Efficient evolution of human antibodies from general protein language models","author":"BL Hie","year":"2023","journal-title":"Nature Biotechnology"},{"key":"pcbi.1012646.ref052","article-title":"Perfecting antibodies with language models","author":"C Outeiral","year":"2023","journal-title":"Nature Biotechnology"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012646","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T00:00:00Z","timestamp":1734480000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012646","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T13:42:51Z","timestamp":1734529371000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012646"}},"subtitle":[],"editor":[{"given":"Miguel Francisco de Almeida Pereira","family":"de Rocha","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,12,6]]},"references-count":52,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12,6]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012646","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,6]]}}}