{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T03:50:49Z","timestamp":1777261849833,"version":"3.51.4"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1013473","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T00:00:00Z","timestamp":1758844800000}}],"reference-count":39,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T00:00:00Z","timestamp":1757548800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Division of Microbiology and Infectious Diseases, National Institute of Allergy and Infectious Diseases and Pendleton Foundation","award":["AI177683"],"award-info":[{"award-number":["AI177683"]}]},{"name":"Division of Microbiology and Infectious Diseases, National Institute of Allergy and Infectious Diseases and the Pendleton Foundation","award":["AI135995"],"award-info":[{"award-number":["AI135995"]}]},{"name":"Division of Microbiology and Infectious Diseases, National Institute of Allergy and Infectious Diseases and the Pendleton Foundation","award":["AI171438"],"award-info":[{"award-number":["AI171438"]}]},{"name":"Division of Microbiology and Infectious Diseases, National Institute of Allergy and Infectious Diseases and the Pendleton Foundation","award":["AI036214"],"award-info":[{"award-number":["AI036214"]}]},{"name":"Division of Microbiology and Infectious Diseases, National Institute of Allergy and Infectious Diseases and the Pendleton Foundation","award":["AI144462"],"award-info":[{"award-number":["AI144462"]}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>\n                    There is growing interest in pre-training antibody language models (\n                    <jats:bold>AbLMs<\/jats:bold>\n                    ) with a mixture of unpaired and natively paired sequences, seeking to combine the proven benefits of training with natively paired sequences with the massive scale of unpaired antibody sequence datasets. However, given the novelty of this strategy, the field lacks a systematic evaluation of data processing methods and training strategies that maximize the benefits of mixed training data while accommodating the significant imbalance in the size of existing paired and unpaired datasets. Here, we introduce a method of curriculum learning for AbLMs, which facilitates a gradual transition from unpaired to paired sequences during training. We optimize this method and compare it to other data sampling strategies for AbLMs, including a constant mix and a fine-tuning approach. We observe that the curriculum and constant approaches show improved performance compared to the fine-tuning approach in large-scale models, likely due to their ability to prevent catastrophic forgetting and slow overfitting. Finally, we show that a 650M-parameter curriculum model, CurrAb, outperforms existing mixed AbLMs in downstream residue prediction and classification tasks.\n                  <\/jats:p>","DOI":"10.1371\/journal.pcbi.1013473","type":"journal-article","created":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T17:43:18Z","timestamp":1757612598000},"page":"e1013473","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":2,"title":["A curriculum learning approach to training antibody language models"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8929-5727","authenticated-orcid":true,"given":"Sarah M.","family":"Burbach","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9535-2866","authenticated-orcid":true,"given":"Bryan","family":"Briney","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2025,9,11]]},"reference":[{"issue":"7744","key":"pcbi.1013473.ref001","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1038\/s41586-019-0879-y","article-title":"Commonality despite exceptional diversity in the baseline human antibody repertoire","volume":"566","author":"B Briney","year":"2019","journal-title":"Nature"},{"issue":"5909","key":"pcbi.1013473.ref002","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1038\/302575a0","article-title":"Somatic generation of antibody diversity","volume":"302","author":"S Tonegawa","year":"1983","journal-title":"Nature"},{"key":"pcbi.1013473.ref003","article-title":"Deciphering antibody affinity maturation with language models and weakly supervised learning","author":"JA Ruffolo","year":"2021"},{"issue":"5","key":"pcbi.1013473.ref004","doi-asserted-by":"crossref","first-page":"100967","DOI":"10.1016\/j.patter.2024.100967","article-title":"Improving antibody language models with native pairing","volume":"5","author":"SM Burbach","year":"2024","journal-title":"Patterns (N Y)"},{"issue":"10","key":"pcbi.1013473.ref005","doi-asserted-by":"crossref","DOI":"10.1016\/j.immuni.2024.07.022","article-title":"An explainable language model for antibody specificity prediction using curated influenza hemagglutinin antibodies","volume":"57","author":"Y Wang","year":"2024","journal-title":"Immunity"},{"key":"pcbi.1013473.ref006","article-title":"Addressing the antibody germline bias and its effect on language models for improved antibody design","author":"TH Olsen","year":"2024","journal-title":"bioRxiv"},{"key":"pcbi.1013473.ref007","article-title":"Scaling laws for neural language models","author":"J Kaplan","year":"2020","journal-title":"arXiv"},{"key":"pcbi.1013473.ref008","article-title":"Training Compute-Optimal Large Language Models","author":"J Hoffmann","year":"2022","journal-title":"arXiv"},{"key":"pcbi.1013473.ref009","article-title":"Large scale paired antibody language models","author":"H Kenlay","year":"2024","journal-title":"arXiv"},{"key":"pcbi.1013473.ref010","article-title":"Enhancing antibody language models with structural information","author":"J Barton","year":"2024","journal-title":"bioRxiv"},{"issue":"13","key":"pcbi.1013473.ref011","doi-asserted-by":"crossref","first-page":"3521","DOI":"10.1073\/pnas.1611835114","article-title":"Overcoming catastrophic forgetting in neural networks","volume":"114","author":"J Kirkpatrick","year":"2017","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pcbi.1013473.ref012","first-page":"6","article-title":"Curriculum learning.","author":"Y Bengio","year":"2009","journal-title":"Proceedings of the 26th Annual International Conference on Machine Learning (ACM)"},{"key":"pcbi.1013473.ref013","article-title":"Curriculum Learning: A Survey","author":"P Soviany","year":"2021","journal-title":"arXiv"},{"key":"pcbi.1013473.ref014","first-page":"751","article-title":"From baby steps to leapfrog: How \u201cless is more\u201d in unsupervised dependency parsing","author":"VI Spitkovsky","year":"2010","journal-title":"North Am Chapter Assoc Comput Linguistics"},{"issue":"1","key":"pcbi.1013473.ref015","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s00354-022-00198-8","article-title":"Length-Based Curriculum Learning for Efficient Pre-training of Language Models","volume":"41","author":"K Nagatsuka","year":"2022","journal-title":"New Gener Comput"},{"issue":"6637","key":"pcbi.1013473.ref016","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Z Lin","year":"2023","journal-title":"Science"},{"key":"pcbi.1013473.ref017","article-title":"RoFormer: Enhanced transformer with Rotary Position Embedding","author":"J Su","year":"2021","journal-title":"arXiv"},{"key":"pcbi.1013473.ref018","article-title":"MiniCPM: Unveiling the potential of Small Language Models with scalable training strategies","author":"S Hu","year":"2024","journal-title":"arXiv"},{"key":"pcbi.1013473.ref019","article-title":"SGDR: Stochastic gradient descent with warm restarts","author":"I Loshchilov","year":"2016","journal-title":"arXiv"},{"issue":"6","key":"pcbi.1013473.ref020","doi-asserted-by":"crossref","first-page":"101239","DOI":"10.1016\/j.patter.2025.101239","article-title":"Focused learning by antibody language models using preferential masking of non-templated regions","volume":"6","author":"K Ng","year":"2025","journal-title":"Patterns (N Y)"},{"issue":"1","key":"pcbi.1013473.ref021","doi-asserted-by":"crossref","DOI":"10.1093\/bioadv\/vbac046","article-title":"AbLang: an antibody language model for completing antibody sequences","volume":"2","author":"TH Olsen","year":"2022","journal-title":"Bioinform Adv"},{"issue":"7","key":"pcbi.1013473.ref022","doi-asserted-by":"crossref","first-page":"100513","DOI":"10.1016\/j.patter.2022.100513","article-title":"Deciphering the language of antibodies using self-supervised learning","volume":"3","author":"J Leem","year":"2022","journal-title":"Patterns (N Y)"},{"key":"pcbi.1013473.ref023","first-page":"506","article-title":"Pre-training Antibody Language Models for Antigen-Specific Computational Antibody Design.","author":"K Gao","year":"2023","journal-title":"Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining KDD\u201923. (Association for Computing Machinery)"},{"key":"pcbi.1013473.ref024","article-title":"Focal Loss for Dense Object Detection","author":"TY Lin","year":"2017","journal-title":"arXiv"},{"key":"pcbi.1013473.ref025","article-title":"P-IgGen: A paired antibody generative language model","author":"OM Turnbull","year":"2024","journal-title":"bioRxiv"},{"key":"pcbi.1013473.ref026","article-title":"A generative foundation model for antibody sequence understanding","author":"J Barton","year":"2024","journal-title":"bioRxiv"},{"issue":"8","key":"pcbi.1013473.ref027","doi-asserted-by":"crossref","first-page":"2502","DOI":"10.4049\/jimmunol.1800708","article-title":"Observed Antibody Space: A Resource for Data Mining Next-Generation Sequencing of Antibody Repertoires","volume":"201","author":"A Kovaltsuk","year":"2018","journal-title":"J Immunol"},{"key":"pcbi.1013473.ref028","article-title":"AntiRef: reference clusters of human antibody sequences","author":"B Briney","year":"2022","journal-title":"bioRxiv"},{"issue":"5","key":"pcbi.1013473.ref029","doi-asserted-by":"crossref","first-page":"734","DOI":"10.1093\/bioinformatics\/btaa739","article-title":"CoV-AbDab: the coronavirus antibody database","volume":"37","author":"MIJ Raybould","year":"2021","journal-title":"Bioinformatics"},{"key":"pcbi.1013473.ref030","doi-asserted-by":"crossref","unstructured":"Jaffe DB, Shahi P, Adams BA, Chrisman AM, Finnegan PM, Raman N, et al. Dataset: Functional antibodies exhibit light chain coherence. 2022. https:\/\/doi.org\/10.5281\/zenodo.6348137","DOI":"10.1101\/2022.04.23.489267"},{"issue":"7","key":"pcbi.1013473.ref031","doi-asserted-by":"crossref","first-page":"1076","DOI":"10.1038\/s41590-022-01230-1","article-title":"Clonal structure, stability and dynamics of human memory B cells and circulating plasmablasts","volume":"23","author":"GE Phad","year":"2022","journal-title":"Nat Immunol"},{"key":"pcbi.1013473.ref032","article-title":"HuggingFace\u2019s Transformers: State-of-the-art Natural Language Processing","author":"T Wolf","year":"2019","journal-title":"arXiv"},{"key":"pcbi.1013473.ref033","first-page":"929","article-title":"PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.","author":"J Ansel","year":"2024","journal-title":"Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2"},{"key":"pcbi.1013473.ref034","article-title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","author":"S Rajbhandari","year":"2019","journal-title":"arXiv"},{"key":"pcbi.1013473.ref035","unstructured":"Gugge S, Debut L, Wolf T, Schmid P, Mueller Z, Mangrulkar S, et al. Accelerate: Training and inference at scale made simple, efficient and adaptable. 2022."},{"key":"pcbi.1013473.ref036","article-title":"Scaling laws and compute-optimal training beyond fixed training durations","author":"A H\u00e4gele","year":"2024","journal-title":"arXiv"},{"key":"pcbi.1013473.ref037","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"issue":"60","key":"pcbi.1013473.ref038","doi-asserted-by":"crossref","first-page":"3021","DOI":"10.21105\/joss.03021","article-title":"seaborn: statistical data visualization","volume":"6","author":"M Waskom","year":"2021","journal-title":"JOSS"},{"issue":"3","key":"pcbi.1013473.ref039","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/MCSE.2007.55","article-title":"Matplotlib: A 2D Graphics Environment","volume":"9","author":"JD Hunter","year":"2007","journal-title":"Comput Sci Eng"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1013473","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T00:00:00Z","timestamp":1758844800000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013473","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T17:52:15Z","timestamp":1758909135000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013473"}},"subtitle":[],"editor":[{"given":"Chaok","family":"Seok","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2025,9,11]]},"references-count":39,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2025,9,11]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1013473","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2025.02.27.640641","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,11]]}}}