{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,11,16]],"date-time":"2026-11-16T18:24:06Z","timestamp":1794853446205,"version":"4.9.1"},"posted":{"date-parts":[[2025,10,23]]},"group-title":"Systems Biology","reference-count":30,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T00:00:00Z","timestamp":1761177600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,10,23]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:p>\n                  Foundation models have transformed natural language processing and computer vision, yet their potential in single-cell biology\u2014particularly for complex diseases such as cancer\u2014 remains underexplored. We present\n                  <jats:monospace>Tahoe-x1<\/jats:monospace>\n                  (Tx1), a family of perturbation-trained single-cell foundation models with up to 3 billion parameters. Tx1 is pretrained on large-scale single-cell transcriptomic datasets, including the\n                  <jats:italic>Tahoe-100M<\/jats:italic>\n                  perturbation compendium, and fine-tuned for cancer-relevant tasks. Through architectural optimizations, data loader refinements, and efficient training strategies, Tx1 achieves 3\u201330\u00d7 higher compute efficiency than prior implementations of cell-state models. Tx1 jointly learns representations of genes, cells, and compounds using a masked-expression generative objective that incorporates a drug token, enabling flexible adaptation to diverse downstream applications. We evaluate Tx1 across four key disease-relevant benchmarks: (1) prediction of overall and context-specific gene essentiality, (2) identification of genes contributing to the hallmarks of cancer, (3) cell-type classification, and (4) prediction of perturbation responses in held-out cellular contexts. Tx1 achieves state-of-the-art performance across all tasks. We release pretrained checkpoints, training code, and evaluation workflows to accelerate the development of perturbation-trained single-cell foundation models for applications in precision oncology and beyond.\n                <\/jats:p>\n                <jats:p>\n                  <jats:fig id=\"ufig1\" position=\"float\" fig-type=\"figure\" orientation=\"portrait\">\n                    <jats:graphic xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"683759v1_ufig1\" position=\"float\" orientation=\"portrait\"\/>\n                  <\/jats:fig>\n                <\/jats:p>","DOI":"10.1101\/2025.10.23.683759","type":"posted-content","created":{"date-parts":[[2025,10,24]],"date-time":"2025-10-24T01:10:23Z","timestamp":1761268223000},"source":"Crossref","is-referenced-by-count":21,"title":["Tahoe-x1: Scaling Perturbation-Trained Single-Cell Foundation Models to 3 Billion Parameters"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-2625-4575","authenticated-orcid":false,"given":"Shreshth","family":"Gandhi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-7415-6825","authenticated-orcid":false,"given":"Farnoosh","family":"Javadi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9217-2330","authenticated-orcid":false,"given":"Valentine","family":"Svensson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6361-4996","authenticated-orcid":false,"given":"Umair","family":"Khan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0363-4493","authenticated-orcid":false,"given":"Matthew G.","family":"Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9857-4349","authenticated-orcid":false,"given":"John","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3728-4401","authenticated-orcid":false,"given":"Daniele","family":"Merico","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9648-8949","authenticated-orcid":false,"given":"Hani","family":"Goodarzi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nima","family":"Alidoust","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2025102704450681000_2025.10.23.683759v1.1","first-page":"121202","article-title":"Human Cell Atlas Meeting Participants: The human cell atlas","volume":"6","year":"2017","journal-title":"Elife"},{"key":"2025102704450681000_2025.10.23.683759v1.2","doi-asserted-by":"publisher","DOI":"10.1101\/2023.04.30.538439"},{"key":"2025102704450681000_2025.10.23.683759v1.3","unstructured":"Xiong, R. , Yang, Y. , He, D. , Zheng, K. , Zheng, S. , Xing, C. , Zhang, H. , Lan, Y. , Wang, L. , Liu, T.-Y. : On Layer Normalization in the Transformer Architecture (2020). https:\/\/arxiv.org\/abs\/2002.04745"},{"key":"2025102704450681000_2025.10.23.683759v1.4","unstructured":"Portes, J. , Trott, A.R. , Havens, S. , King, D. , Venigalla, A. , Nadeem, M. , Sardana, N. , Khudia, D. , Frankle, J. : Mosaicbert: A bidirectional encoder optimized for fast pretraining. NeurIPS \/ OpenReview (2023). Mentions use of low precision LayerNorm in architecture"},{"key":"2025102704450681000_2025.10.23.683759v1.5","unstructured":"Team, T.M.M. : streaming. <https:\/\/github.com\/mosaicml\/streaming\/> (2022)"},{"key":"2025102704450681000_2025.10.23.683759v1.6","doi-asserted-by":"crossref","unstructured":"Zhao, Y. , Gu, A. , Varma, R. , Luo, L. , Huang, C.-C. , Xu, M. , Wright, L. , Shojanazeri, H. , Ott, M. , Shleifer, S. , Desmaison, A. , Balioglu, C. , Damania, P. , Nguyen, B. , Chauhan, G. , Hao, Y. , Mathews, A. , Li, S. : PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023). https:\/\/arxiv.org\/abs\/2304.11277","DOI":"10.14778\/3611540.3611569"},{"key":"2025102704450681000_2025.10.23.683759v1.7","doi-asserted-by":"crossref","unstructured":"Youngblut, N.D. , Carpenter, C. , Prashar, J. , Ricci-Tam, C. , Ilango, R. , Teyssier, N. , Konermann, S. , Hsu, P.D. , Dobin, A. , Burke, D.P. , Goodarzi, H. , Roohani, Y.H. : scBaseCount: an AI agentcurated, uniformly processed, and continually expanding single cell data repository. bioRxiv, 2025\u20130227640494 (2025)","DOI":"10.1101\/2025.02.27.640494"},{"key":"2025102704450681000_2025.10.23.683759v1.8","unstructured":"CZI Single-Cell Biology Program, Abdulla, S. , Aevermann, B. , Assis, P. , Badajoz, S. , Bell, S.M. , Bezzi, E. , Cakir, B. , Chaffer, J. , Chambers, S. , Michael Cherry, J. , Chi, T. , Chien, J. , Dorman, L. , Garcia-Nieto, P. , Gloria, N. , Hastie, M. , Hegeman, D. , Hilton, J. , Huang, T. , Infeld, A. , Istrate, A.-M. , Jelic, I. , Katsuya, K. , Kim, Y.J. , Liang, K. , Lin, M. , Lombardo, M. , Marshall, B. , Martin, B. , McDade, F. , Megill, C. , Patel, N. , Predeus, A. , Raymor, B. , Robatmili, B. , Rogers, D. , Rutherford, E. , Sadgat, D. , Shin, A. , Small, C. , Smith, T. , Sridharan, P. , Tarashansky, A. , Tavares, N. , Thomas, H. , Tolopko, A. , Urisko, M. , Yan, J. , Yeretssian, G. , Zamanian, J. , Mani, A. , Cool, J. , Carr, A. : CZ CELL\u00d7GENE discover: A single-cell data platform for scalable exploration, analysis and modeling of aggregated data. bioRxiv, 2023\u20131030563174 (2023)"},{"key":"2025102704450681000_2025.10.23.683759v1.9","doi-asserted-by":"publisher","DOI":"10.1038\/s41568-024-00763-x"},{"key":"2025102704450681000_2025.10.23.683759v1.10","doi-asserted-by":"publisher","DOI":"10.1038\/ng.3984"},{"key":"2025102704450681000_2025.10.23.683759v1.11","doi-asserted-by":"publisher","DOI":"10.1038\/nature11003"},{"key":"2025102704450681000_2025.10.23.683759v1.12","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0506580102"},{"issue":"6","key":"2025102704450681000_2025.10.23.683759v1.13","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1016\/j.cels.2015.12.004","article-title":"The molecular signatures database hallmark gene set collection","volume":"1","year":"2015","journal-title":"Cell systems"},{"key":"2025102704450681000_2025.10.23.683759v1.14","unstructured":"Tabula Sapiens Consortium, Quake, S.R. : Tabula sapiens reveals transcription factor expression, senescence effects, and sex-specific features in cell types from 28 human organs and tissues. bioRxiv (2025)"},{"key":"2025102704450681000_2025.10.23.683759v1.15","doi-asserted-by":"publisher","DOI":"10.1101\/2025.06.26.661135"},{"key":"2025102704450681000_2025.10.23.683759v1.16","doi-asserted-by":"publisher","DOI":"10.1101\/2023.11.27.567555"},{"key":"2025102704450681000_2025.10.23.683759v1.17","unstructured":"Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A.N. , Kaiser, L. , Polosukhin, I. : Attention Is All You Need (2023). https:\/\/arxiv.org\/abs\/1706.03762"},{"key":"2025102704450681000_2025.10.23.683759v1.18","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. , Toutanova, K. : BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019). https:\/\/arxiv.org\/abs\/1810.04805"},{"key":"2025102704450681000_2025.10.23.683759v1.19","unstructured":"Liu, Y. , Ott, M. , Goyal, N. , Du, J. , Joshi, M. , Chen, D. , Levy, O. , Lewis, M. , Zettlemoyer, L. , Stoyanov, V. : RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). https:\/\/arxiv.org\/abs\/1907.11692"},{"key":"2025102704450681000_2025.10.23.683759v1.20","doi-asserted-by":"publisher","DOI":"10.1101\/2025.02.20.639398"},{"key":"2025102704450681000_2025.10.23.683759v1.21","doi-asserted-by":"publisher","DOI":"10.1101\/2025.02.27.640494"},{"key":"2025102704450681000_2025.10.23.683759v1.22","doi-asserted-by":"publisher","DOI":"10.1101\/2023.10.30.563174"},{"key":"2025102704450681000_2025.10.23.683759v1.23","unstructured":"Micikevicius, P. , Narang, S. , Alben, J. , Diamos, G. , Elsen, E. , Garcia, D. , Ginsburg, B. , Houston, M. , Kuchaiev, O. , Venkatesh, G. , Wu, H. : Mixed Precision Training (2018). https:\/\/arxiv.org\/abs\/1710.03740"},{"key":"2025102704450681000_2025.10.23.683759v1.24","unstructured":"Radford, A. , Wu, J. , Child, R. , Luan, D. , Amodei, D. , Sutskever, I. : Language models are unsupervised multitask learners. (2019). https:\/\/api.semanticscholar.org\/CorpusID:160025533"},{"key":"2025102704450681000_2025.10.23.683759v1.25","unstructured":"Loshchilov, I. , Hutter, F. : Decoupled Weight Decay Regularization (2019). https:\/\/arxiv.org\/abs\/1711.05101"},{"key":"2025102704450681000_2025.10.23.683759v1.26","unstructured":"Dao, T. : FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023). https:\/\/arxiv.org\/abs\/2307.08691"},{"key":"2025102704450681000_2025.10.23.683759v1.27","unstructured":"Shah, J. , Bikshandi, G. , Zhang, Y. , Thakkar, V. , Ramani, P. , Dao, T. : FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024). https:\/\/arxiv.org\/abs\/2407.08608"},{"key":"2025102704450681000_2025.10.23.683759v1.28","first-page":"26736","article-title":"The stability-efficiency dilemma: Investigating sequence length warmup for training gpt models","volume":"35","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2025102704450681000_2025.10.23.683759v1.29","unstructured":"Jin, H. , Han, X. , Yang, J. , Jiang, Z. , Chang, C.-Y. , Hu, X. : GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length (2023). https:\/\/arxiv.org\/abs\/2310.00576"},{"key":"2025102704450681000_2025.10.23.683759v1.30","doi-asserted-by":"crossref","unstructured":"Virshup, I. , Rybakov, S. , Theis, F.J. , Angerer, P. , Wolf, F.A. : anndata: Annotated data. bioRxiv (2021)","DOI":"10.1101\/2021.12.16.473007"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.10.23.683759","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T09:52:41Z","timestamp":1768470761000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.1101\/2025.10.23.683759"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,23]]},"references-count":30,"URL":"https:\/\/doi.org\/10.1101\/2025.10.23.683759","relation":{},"subject":[],"published":{"date-parts":[[2025,10,23]]},"subtype":"preprint"}}