{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T01:58:51Z","timestamp":1791597531909,"version":"4.3.4"},"reference-count":63,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"37","funder":[{"name":"Major Project of Guangzhou National Laboratory","award":["GZNL2024A01015"],"award-info":[{"award-number":["GZNL2024A01015"]}]}],"content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Adv."],"published-print":{"date-parts":[[2026,9,11]]},"abstract":"<jats:p>Predicting single-cell transcriptional responses to perturbations is central to dissecting gene regulation and accelerating therapeutic design, yet the field lacks a rigorous, task-spanning assessment of model behavior. We present a large-scale benchmark of 13 representative methods and baselines across 25 datasets spanning diverse perturbation modalities and species, including two primary immune-cell drug-response resources. We evaluated three core tasks\u2014generalization to unseen single-gene perturbations, prediction of combinatorial interactions, and transfer across cell types\u2014using 24 metrics covering expression-level accuracy, relative changes, differential expression (DE) recovery, and distributional similarity. Across tasks, performance depended strongly on perturbation effect size and evaluation perspective: Expression-level agreement was the highest for small-effect perturbations resembling controls, whereas delta- and DE-based metrics improved with larger effects, providing clearer signals. Models shared a conservative bias, with fine-tuned foundation models compressing variance and underestimating synergistic effects in combinations. PerturbNet showed superior recovery of DE signatures in Tasks 1 and 2, while no method consistently generalized across cell types in Task 3, where biological consistency dominated outcomes. This benchmark establishes current methodological limits, clarifies that different metrics probe distinct biological signals rather than redundant summaries of the same prediction problem, and provides a foundation for developing virtual-cell models that more faithfully capture heterogeneous perturbation responses.<\/jats:p>","DOI":"10.1126\/sciadv.aed3414","type":"journal-article","created":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T17:58:47Z","timestamp":1788976727000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":1,"title":["A systematic comparison of single-cell perturbation response prediction models"],"prefix":"10.1126","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2921-9105","authenticated-orcid":true,"given":"Lanxiang","family":"Li","sequence":"first","affiliation":[{"name":"GMU-GIBH Joint School of Life Sciences, Guangzhou Medical University, Guangzhou, China."},{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3883-445X","authenticated-orcid":true,"given":"Yue","family":"You","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8292-3169","authenticated-orcid":true,"given":"Yunlin","family":"Fu","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenyu","family":"Liao","sequence":"additional","affiliation":[{"name":"The University of Hong Kong, Hong Kong, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xueying","family":"Fan","sequence":"additional","affiliation":[{"name":"School of Life Sciences, Westlake University, Hangzhou, China."},{"name":"Research Center for Industries of the Future, Westlake University, Hangzhou, China."},{"name":"Westlake Laboratory of Life Sciences and Biomedicine, Hangzhou, China."},{"name":"Westlake Institute for Advanced Study, Hangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3787-0063","authenticated-orcid":true,"given":"Shihong","family":"Lu","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4999-2185","authenticated-orcid":true,"given":"Ye","family":"Cao","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bo","family":"Li","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5376-2634","authenticated-orcid":true,"given":"Wenle","family":"Ren","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaming","family":"Kong","sequence":"additional","affiliation":[{"name":"Renaissance Era Innovation Technology Co., Ltd., Beijing, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9747-4285","authenticated-orcid":true,"given":"Shuangjia","family":"Zheng","sequence":"additional","affiliation":[{"name":"Global Institute of Future Technology, Shanghai Jiao Tong University, Shanghai, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jizheng","family":"Chen","sequence":"additional","affiliation":[{"name":"Guangzhou National Laboratory, Guangzhou, China."},{"name":"State Key Laboratory of Respiratory Disease, National Clinical Research Center for Respiratory Disease, Guangzhou Institute of Respiratory Health, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9315-3406","authenticated-orcid":true,"given":"Xiaodong","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Life Sciences, Westlake University, Hangzhou, China."},{"name":"Research Center for Industries of the Future, Westlake University, Hangzhou, China."},{"name":"Westlake Laboratory of Life Sciences and Biomedicine, Hangzhou, China."},{"name":"Westlake Institute for Advanced Study, Hangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3420-3685","authenticated-orcid":true,"given":"Luyi","family":"Tian","sequence":"additional","affiliation":[{"name":"GMU-GIBH Joint School of Life Sciences, Guangzhou Medical University, Guangzhou, China."},{"name":"Guangzhou National Laboratory, Guangzhou, China."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-08338-4"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00395-022-00972-1"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2016.11.038"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.4177"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-10216-x"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1038\/ng.3314"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41579-020-00440-4"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-023-02144-y"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2024.11.015"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2024.10.029"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1093\/bfgp\/elx046"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.15252\/msb.20199198"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cels.2021.05.016"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-023-01905-6"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-024-02201-0"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2024.07.035"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2024.04.058"},{"key":"e_1_3_2_19_2","unstructured":"J. Devlin M.-W. Chang K. Lee K. Toutanova BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 [cs.CL] (2019); https:\/\/arxiv.org\/abs\/1810.04805."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06139-9"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkab775"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-024-02305-7"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"A. Wenteler M. Occhetta N. Branson M. Huebner V. Curean W. T. Dee W. T. Connell A. Hawkins-Hooker S. P. Chung Y. Ektefaie A. Gallagher-Syed C. M. V. C\u00f3rdova PertEval-scFM: Benchmarking single-cell foundation models for perturbation effect prediction. bioRxiv 616248 (2024); https:\/\/doi.org\/10.1101\/2024.10.02.616248.","DOI":"10.1101\/2024.10.02.616248"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"Y. Wu E. Wershof S. M. Schmon M. Nassar B. Osi\u0144ski R. Eksi Z. Yan R. Stark K. Zhang T. Graepel Perturbench: Benchmarking machine learning models for cellular perturbation analysis. arXiv:2408.10609 [cs.LG] (2024).","DOI":"10.52202\/085713-3225"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-025-03574-x"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-025-02772-6"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-025-02777-8"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2025.06.008"},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"A. K. Adduri D. Gautam B. Bevilacqua A. Imran R. Shah M. Naghipourfar N. Teyssier R. Ilango S. Nagaraj M. Dong C. Ricci-Tam C. Carpenter V. Subramanyam A. Winters S. Tirukkovular J. Sullivan B. S. Plosky B. Eraslan N. D. Youngblut J. Leskovec L. A. Gilbert S. Konermann P. D. Hsu A. Dobin D. P. Burke H. Goodarzi Y. H. Roohani Predicting cellular responses to perturbation across diverse contexts with STATE. bioRxiv 661135 (2025); https:\/\/doi.org\/10.1101\/2025.06.26.661135.","DOI":"10.1101\/2025.06.26.661135"},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","unstructured":"J. Zhang A. A. Ubas R. de Borja V. Svensson N. Thomas N. Thakar I. Lai A. Winters U. Khan M. G. Jones J. D. Thompson V. Tran J. Pangallo E. Papalexi A. Sapre H. Nguyen O. Sanderson M. Nigos O. Kaplan S. Schroeder B. Hariadi S. Marrujo C. A. S. Curca G. Gallareta Olivares R. Koehler G. Geiss A. Rosenberg C. Roco D. Merico N. Alidoust H. Goodarzi J. Yu Tahoe-100M: A giga-scale single-cell perturbation atlas for context-dependent gene function and cellular modeling. bioRxiv 639398 (2025); https:\/\/doi.org\/10.1101\/2025.02.20.639398.","DOI":"10.1101\/2025.02.20.639398"},{"key":"e_1_3_2_31_2","unstructured":"G. M. Mejia H. E. Miller F. J. Leblanc B. Wang B. Swain L. P. d. L. Camillo Diversity by design: Addressing mode collapse improves scRNA-seq perturbation modeling on well-calibrated metrics. arXiv:2506.22641 [q-bio.GN] (2025)."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-025-02980-0"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1038\/s44320-025-00131-3"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","unstructured":"H. C. Moon M. H. Herschl A. Pawluk S. Konermann P. D. Hsu A combinatorial domain screening platform reveals epigenetic effector interactions for transcriptional perturbation. bioRxiv 620683 (2024); https:\/\/doi.org\/10.1101\/2024.10.28.620683.","DOI":"10.1101\/2024.10.28.620683"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1002\/advs.202404845"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"R. A. Saunders W. E. Allen X. Pan J. Sandhu J. Lu T. K. Lau K. Smolyar Z. A. Sullivan C. Dulac J. S. Weissman X. Zhuang A platform for multimodal in vivo pooled genetic screens reveals regulators of liver function. bioRxiv 624217 (2025); https:\/\/doi.org\/10.1101\/2024.11.18.624217.","DOI":"10.1101\/2024.11.18.624217"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cels.2024.12.002"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-021-01153-z"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41588-021-00779-1"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41588-021-00778-2"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2022.05.013"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-021-00862-0"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aax6234"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-023-41788-4"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aax4438"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2016.11.048"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.4042"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature24489"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-018-0657-2"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaw3381"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-023-01969-x"},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","unstructured":"J.-L. Liu L.-G. Yang Q.-Y. Xiao Z.-Q. Li J.-Y. Cheng Z.-R. Chen J.-W. Zhou X. Wan T.-H. Chang X. Zhang Y.-X. Li Bio-medical big data Operating System (bio-OS): An integrated data mining environment for data intensive scientific research. bioRxiv 612997 (2024); https:\/\/doi.org\/10.1101\/2024.10.17.612997.","DOI":"10.1101\/2024.10.17.612997"},{"key":"e_1_3_2_53_2","unstructured":"F. Wenkel W. Tu C. Masschelein H. Shirzad C. Eastwood S. T. Whitfield I. Bendidi C. Russell L. Hodgson Y. El Mesbahi J. Ding M. M. Fay B. Earnshaw E. Noutahi A. K. Denton TxPert: Leveraging biochemical relationships for out-of-distribution transcriptomic perturbation prediction. arXiv:2505.14919 [cs.LG] (2025)."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-025-03840-y"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btaf317"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-019-0494-8"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btaa800"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2023.100817"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btae265"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.15252\/msb.202211517"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btac357"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-023-02079-x"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-025-02877-y"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41588-025-02169-3"}],"container-title":["Science Advances"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/sciadv.aed3414","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T17:59:45Z","timestamp":1788976785000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/sciadv.aed3414"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,11]]},"references-count":63,"journal-issue":{"issue":"37","published-print":{"date-parts":[[2026,9,11]]}},"alternative-id":["10.1126\/sciadv.aed3414"],"URL":"https:\/\/doi.org\/10.1126\/sciadv.aed3414","relation":{},"ISSN":["2375-2548"],"issn-type":[{"value":"2375-2548","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,11]]},"assertion":[{"value":"2025-10-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eaed3414"}}