{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T13:07:44Z","timestamp":1784552864529,"version":"3.55.0"},"posted":{"date-parts":[[2026,7,16]]},"group-title":"Immunology","reference-count":59,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T00:00:00Z","timestamp":1784160000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"crossref","award":["75N93024R00017"],"award-info":[{"award-number":["75N93024R00017"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"crossref","award":["1U24AI177622"],"award-info":[{"award-number":["1U24AI177622"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"crossref"}]},{"name":"UIOLifeScience convergence environment \u201cUiORealArt\u201c"},{"name":"Norwegian Cancer Society Grant","award":["215817"],"award-info":[{"award-number":["215817"]}]},{"name":"Research Council of Norway projects","award":["300740"],"award-info":[{"award-number":["300740"]}]},{"name":"Research Council of Norway projects","award":["331890"],"award-info":[{"award-number":["331890"]}]},{"name":"European Research Council Consolidator Grant AB-AG-INTERACT","award":["101125630"],"award-info":[{"award-number":["101125630"]}]},{"name":"Innovative Medicines Initiative 2 Joint Undertaking","award":["101007799"],"award-info":[{"award-number":["101007799"]}]},{"name":"UIOLifeScience internationalization support"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,7,16]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>Generative models are increasingly used to model adaptive immune receptor repertoire (AIRR) sequence distributions, promising to decode the sequence diversity shaping immune responses and accelerate the design of therapeutic antibodies and T-cell receptors. Yet it remains unclear whether these models produce biologically meaningful outputs or merely capture surface-level sequence statistics while missing features driven by receptor generation and selection. Rigorous evaluation is needed, but the field lacks established standards, as existing machine learning metrics do not all translate directly to the AIRR domain, given the complex structure of the data and the lack of biological ground truth. Consequently, researchers face difficulties in evaluating the models and selecting appropriate ones, which can critically affect downstream clinical applications.<\/jats:p>\n                <jats:p>Here, we apply a suite of evaluation metrics tailored to AIRR sequence data and present a systematic comparison of popular generative model families proposed for the AIRR field, including variational autoencoders, long short-term memory networks, antibody language models, selection models, and simple statistical baselines. We focus specifically on the task of learning individual-specific immune receptor repertoires, a clinically relevant challenge with direct implications for personalized immunotherapy, disease monitoring, and vaccine response studies. By analyzing the sequences generated by each model, we identify memorization risks, innovation capabilities, and sensitivity to hyperparameter tuning. Taken together, these results advance the understanding of how current generative models reproduce the biology of individual immune repertoires and lay the groundwork for more principled model development and evaluation.<\/jats:p>","DOI":"10.64898\/2026.07.10.737788","type":"posted-content","created":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T23:40:12Z","timestamp":1784245212000},"source":"Crossref","is-referenced-by-count":0,"title":["What Do Generative Models Learn About Adaptive Immune Receptor Repertoires? A Benchmark Study"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7472-6985","authenticated-orcid":false,"given":"Charlotte","family":"W\u00fcrtzen","sequence":"first","affiliation":[{"name":"Department of Informatics, University of Oslo, Norway;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7846-4897","authenticated-orcid":false,"given":"Maria","family":"Mamica","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Oslo, Norway;"},{"name":"UiORealArt Convergence Environment, University of Oslo, Norway;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4783-9060","authenticated-orcid":false,"given":"Chakravarthi","family":"Kanduri","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Oslo, Norway;"},{"name":"UiORealArt Convergence Environment, University of Oslo, Norway;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2484-3868","authenticated-orcid":false,"given":"Milena","family":"Pavlovi\u0107","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Oslo, Norway;"},{"name":"UiORealArt Convergence Environment, University of Oslo, Norway;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2622-5032","authenticated-orcid":false,"given":"Victor","family":"Greiff","sequence":"additional","affiliation":[{"name":"Department of Immunology, University of Oslo and Oslo University Hospital, Oslo, Norway;"},{"name":"Imprint Labs, LLC. New York, NY, USA;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8457-6693","authenticated-orcid":false,"given":"Bjoern","family":"Peters","sequence":"additional","affiliation":[{"name":"Center for Vaccine Innovation, La Jolla Institute for Immunology, La Jolla, San Diego, CA, USA;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4959-1409","authenticated-orcid":false,"given":"Geir Kjetil","family":"Sandve","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Oslo, Norway;"},{"name":"UiORealArt Convergence Environment, University of Oslo, Norway;"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026072006001643000_2026.07.10.737788v1.1","doi-asserted-by":"publisher","DOI":"10.1038\/s43586-023-00284-1"},{"key":"2026072006001643000_2026.07.10.737788v1.2","doi-asserted-by":"publisher","DOI":"10.1038\/302575a0"},{"key":"2026072006001643000_2026.07.10.737788v1.3","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-immunol-042718-041757"},{"key":"2026072006001643000_2026.07.10.737788v1.4","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-chembioeng-101420-125021"},{"key":"2026072006001643000_2026.07.10.737788v1.5","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-02832-w"},{"key":"2026072006001643000_2026.07.10.737788v1.6","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz035"},{"key":"2026072006001643000_2026.07.10.737788v1.7","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1008394"},{"key":"2026072006001643000_2026.07.10.737788v1.8","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2023141118"},{"key":"2026072006001643000_2026.07.10.737788v1.9","doi-asserted-by":"publisher","DOI":"10.1126\/science.ads0018"},{"key":"2026072006001643000_2026.07.10.737788v1.10","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-48198-0"},{"key":"2026072006001643000_2026.07.10.737788v1.11","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbad038"},{"key":"2026072006001643000_2026.07.10.737788v1.12","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-49806-9"},{"key":"2026072006001643000_2026.07.10.737788v1.13","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.46935"},{"key":"2026072006001643000_2026.07.10.737788v1.14","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-85274-7"},{"key":"2026072006001643000_2026.07.10.737788v1.15","doi-asserted-by":"publisher","DOI":"10.1080\/19420862.2022.2031482"},{"key":"2026072006001643000_2026.07.10.737788v1.16","doi-asserted-by":"publisher","DOI":"10.1016\/j.cels.2023.10.002"},{"key":"2026072006001643000_2026.07.10.737788v1.17","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkaf025"},{"key":"2026072006001643000_2026.07.10.737788v1.18","doi-asserted-by":"publisher","DOI":"10.1101\/2020.04.12.024844"},{"key":"2026072006001643000_2026.07.10.737788v1.19","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-01172-2"},{"key":"2026072006001643000_2026.07.10.737788v1.20","first-page":"9754","article-title":"Antigen-Specific Antibody Design and Optimization with Diffusion-Based Generative Models for Protein Structures","volume":"35","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026072006001643000_2026.07.10.737788v1.21","doi-asserted-by":"publisher","DOI":"10.1016\/j.cels.2023.10.001"},{"key":"2026072006001643000_2026.07.10.737788v1.22","unstructured":"Murphy K. Probabilistic Machine Learning: Advanced Topics [Internet]. MIT Press; 2023 [cited 2025 Feb 20]. Available from: https:\/\/probml.github.io\/pml-book\/book2.html"},{"key":"2026072006001643000_2026.07.10.737788v1.23","first-page":"34773","article-title":"A Geometric Framework for Understanding Memorization in Generative Models","volume":"2025","year":"2025","journal-title":"Int Conf Learn Represent"},{"key":"2026072006001643000_2026.07.10.737788v1.24","unstructured":"van den Burg G , Williams C. On Memorization in Probabilistic Deep Generative Models. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2021 [cited 2025 Sep 4]. p. 27916\u201328. Available from: https:\/\/proceedings.neurips.cc\/paper\/2021\/hash\/eae15aabaa768ae4a5993a8a4f4fa6e4-Abstract.html"},{"key":"2026072006001643000_2026.07.10.737788v1.25","unstructured":"Salimans T , Goodfellow I , Zaremba W , Cheung V , Radford A , Chen X , et al. Improved Techniques for Training GANs. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2016 [cited 2025 Sep 12]. Available from: https:\/\/papers.nips.cc\/paper_files\/paper\/2016\/hash\/8a3363abe792db2d8761d6403605aeb7-Abstract.html"},{"key":"2026072006001643000_2026.07.10.737788v1.26","unstructured":"Heusel M , Ramsauer H , Unterthiner T , Nessler B , Hochreiter S. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2017 [cited 2025 Sep 12]. Available from: https:\/\/papers.nips.cc\/paper_files\/paper\/2017\/hash\/8a1d694707eb0fefe65871369074926d-Abstract.html"},{"key":"2026072006001643000_2026.07.10.737788v1.27","unstructured":"Sajjadi MSM , Bachem O , Lucic M , Bousquet O , Gelly S. Assessing Generative Models via Precision and Recall. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2018 [cited 2026 Jul 9]. Available from: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2018\/hash\/f7696a9b362ac5a51c3dc8f098b73923-Abstract.html"},{"key":"2026072006001643000_2026.07.10.737788v1.28","unstructured":"Kynk\u00e4\u00e4nniemi T , Karras T , Laine S , Lehtinen J , Aila T. Improved Precision and Recall Metric for Assessing Generative Models. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2019 [cited 2026 Jul 9]. Available from: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/hash\/0234c510bc6d908b28c70ff313743079-Abstract.html"},{"key":"2026072006001643000_2026.07.10.737788v1.29","unstructured":"Naeem MF , Oh SJ , Uh Y , Choi Y , Yoo J. Reliable Fidelity and Diversity Metrics for Generative Models. In: Proceedings of the 37th International Conference on Machine Learning [Internet]. PMLR; 2020 [cited 2026 Jul 9]. p. 7176\u201385. Available from: https:\/\/proceedings.mlr.press\/v119\/naeem20a.html"},{"key":"2026072006001643000_2026.07.10.737788v1.30","unstructured":"Khayatkhoei M , Abdalmageed W. Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions. In: Proceedings of the 40th International Conference on Machine Learning [Internet]. PMLR; 2023 [cited 2025 Apr 24]. p. 16326\u201343. Available from: https:\/\/proceedings.mlr.press\/v202\/khayatkhoei23a.html"},{"key":"2026072006001643000_2026.07.10.737788v1.31","unstructured":"Alaa A , Breugel BV , Saveliev ES , Schaar M van der. How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models. In: Proceedings of the 39th International Conference on Machine Learning [Internet]. PMLR; 2022 [cited 2025 Apr 24]. p. 290\u2013306. Available from: https:\/\/proceedings.mlr.press\/v162\/alaa22a.html"},{"key":"2026072006001643000_2026.07.10.737788v1.32","doi-asserted-by":"publisher","DOI":"10.3389\/fimmu.2022.858057"},{"key":"2026072006001643000_2026.07.10.737788v1.33","doi-asserted-by":"publisher","DOI":"10.1016\/j.cels.2024.11.006"},{"key":"2026072006001643000_2026.07.10.737788v1.34","doi-asserted-by":"publisher","DOI":"10.1038\/s42003-025-07708-6"},{"key":"2026072006001643000_2026.07.10.737788v1.35","doi-asserted-by":"publisher","DOI":"10.1101\/2024.10.25.620213"},{"key":"2026072006001643000_2026.07.10.737788v1.36","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btac353"},{"key":"2026072006001643000_2026.07.10.737788v1.37","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-026-01244-6"},{"key":"2026072006001643000_2026.07.10.737788v1.38","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2024.3514089"},{"key":"2026072006001643000_2026.07.10.737788v1.39","unstructured":"Zhu T , Ren M , Zhang H. Antibody Design Using a Score-based Diffusion Model Guided by Evolutionary, Physical and Geometric Constraints. In: Proceedings of the 41st International Conference on Machine Learning [Internet]. PMLR; 2024 [cited 2025 Sep 3]. p. 62531\u201348. Available from: https:\/\/proceedings.mlr.press\/v235\/zhu24j.html"},{"key":"2026072006001643000_2026.07.10.737788v1.40","doi-asserted-by":"crossref","unstructured":"Alamdari S , Thakkar N , Berg R van den, Lu A , Fusi N , Amini A , et al. Protein generation with evolutionary diffusion: sequence is all you need. In: NeurIPS 2023 Generative AI and Biology (GenBio) Workshop [Internet]. 2023 [cited 2026 Jul 9]. Available from: https:\/\/openreview.net\/forum?id=qP69kXPdJM","DOI":"10.1101\/2023.09.11.556673"},{"key":"2026072006001643000_2026.07.10.737788v1.41","doi-asserted-by":"publisher","DOI":"10.1186\/s13073-015-0169-8"},{"key":"2026072006001643000_2026.07.10.737788v1.42","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.101.062414"},{"key":"2026072006001643000_2026.07.10.737788v1.43","doi-asserted-by":"publisher","DOI":"10.64898\/2026.04.15.718648"},{"key":"2026072006001643000_2026.07.10.737788v1.44","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01153"},{"key":"2026072006001643000_2026.07.10.737788v1.45","unstructured":"Zhang J , Li CT , Farnia F. An Interpretable Evaluation of Entropy-based Novelty of Generative Models. In: Proceedings of the 41st International Conference on Machine Learning [Internet]. PMLR; 2024 [cited 2026 Jun 2]. p. 59148\u201372. Available from: https:\/\/proceedings.mlr.press\/v235\/zhang24ac.html"},{"key":"2026072006001643000_2026.07.10.737788v1.46","unstructured":"Jiralerspong M , Bose J , Gemp I , Qin C , Bachrach Y , Gidel G. Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples. In: Advances in Neural Information Processing Systems [Internet]. Curran Associates, Inc.; 2023 [cited 2026 Jul 9]. p. 33095\u2013119. Available from: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/hash\/68b138608ef80b08d65b1bd9594d9559-Abstract-Conference.html"},{"key":"2026072006001643000_2026.07.10.737788v1.47","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-93158-2"},{"key":"2026072006001643000_2026.07.10.737788v1.48","doi-asserted-by":"publisher","DOI":"10.1103\/PRXLife.2.013011"},{"key":"2026072006001643000_2026.07.10.737788v1.49","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1511.01844"},{"key":"2026072006001643000_2026.07.10.737788v1.50","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2512.09152"},{"key":"2026072006001643000_2026.07.10.737788v1.51","doi-asserted-by":"publisher","DOI":"10.1111\/imr.12666"},{"key":"2026072006001643000_2026.07.10.737788v1.52","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2021.01.050"},{"key":"2026072006001643000_2026.07.10.737788v1.53","doi-asserted-by":"publisher","DOI":"10.1172\/jci.insight.88242"},{"key":"2026072006001643000_2026.07.10.737788v1.54","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-021-00413-z"},{"key":"2026072006001643000_2026.07.10.737788v1.55","doi-asserted-by":"publisher","DOI":"10.1109\/18.61115"},{"key":"2026072006001643000_2026.07.10.737788v1.56","doi-asserted-by":"publisher","DOI":"10.1016\/j.copbio.2020.07.010"},{"key":"2026072006001643000_2026.07.10.737788v1.57","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btac505"},{"key":"2026072006001643000_2026.07.10.737788v1.58","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-09278-8"},{"key":"2026072006001643000_2026.07.10.737788v1.59","unstructured":"Manning C , Raghavan P , Schuetze H . Introduction to Information Retrieval. Cambridge University Press; 2009."}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.07.10.737788","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T13:00:26Z","timestamp":1784552426000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.64898\/2026.07.10.737788"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,16]]},"references-count":59,"URL":"https:\/\/doi.org\/10.64898\/2026.07.10.737788","relation":{},"subject":[],"published":{"date-parts":[[2026,7,16]]},"subtype":"preprint"}}