{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T23:14:30Z","timestamp":1787008470795,"version":"build-2736575974"},"reference-count":27,"publisher":"Oxford University Press (OUP)","issue":"Supplement_1","license":[{"start":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:00:00Z","timestamp":1783382400000},"content-version":"vor","delay-in-days":6,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["92470105"],"award-info":[{"award-number":["92470105"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2025YFC3409300"],"award-info":[{"award-number":["2025YFC3409300"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Tsinghua-Toyota Joint Research Institute Inter-disciplinary Program","award":["20243930093"],"award-info":[{"award-number":["20243930093"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Recent advances in large language models have enabled the emergence of AI scientists that aim to autonomously analyze biological data and assist scientific discovery. Despite rapid progress, it remains unclear to what extent these systems can extract meaningful biological insights from real experimental data. Existing benchmarks either evaluate reasoning in the absence of data or focus on predefined analytical outputs, failing to reflect realistic, data-driven biological research.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Here, we introduce BAISBench (Biological AI Scientist Benchmark), a benchmark for evaluating AI scientists on real single-cell transcriptomic datasets. BAISBench comprises two tasks: cell type annotation across 15 expert-labeled datasets, and scientific discovery through 193 multiple-choice questions derived from biological conclusions reported in 41 published single-cell studies. We evaluated several representative AI scientists using BAISBench and, to provide a human performance baseline, invited five graduate-level bioinformaticians to collectively complete the same tasks. The results show that while current AI scientists fall short of fully autonomous biological discovery, they already demonstrate substantial potential in supporting data-driven biological research. These results position BAISBench as a practical benchmark for characterizing the current capabilities and limitations of AI scientists in biological research. We expect BAISBench to serve as a practical evaluation framework for guiding the development of more capable AI scientists and for helping biologists identify AI systems that can effectively support real-world research workflows.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>https:\/\/github.com\/EperLuo\/BAISBench, https:\/\/huggingface.co\/datasets\/EperLuo\/BaisBench.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btag227","type":"journal-article","created":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T16:31:54Z","timestamp":1783441914000},"source":"Crossref","is-referenced-by-count":2,"title":["Benchmarking AI scientists for omics data\u2013driven biological discovery"],"prefix":"10.1093","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-4087-6027","authenticated-orcid":false,"given":"Erpai","family":"Luo","sequence":"first","affiliation":[{"name":"MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinmeng","family":"Jia","sequence":"additional","affiliation":[{"name":"MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifan","family":"Xiong","sequence":"additional","affiliation":[{"name":"MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6017-0701","authenticated-orcid":false,"given":"Xiangyu","family":"Li","sequence":"additional","affiliation":[{"name":"School of Software Engineering, Beijing Jiaotong University , Beijing, 100044,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaobo","family":"Guo","sequence":"additional","affiliation":[{"name":"Department of Physiology and Pathophysiology, School of Basic Medical Sciences, Capital Medical University , Beijing, 100069,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baoqi","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Physiology and Pathophysiology, School of Basic Medical Sciences, Capital Medical University , Beijing, 100069,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minsheng","family":"Hao","sequence":"additional","affiliation":[{"name":"MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Wei","sequence":"additional","affiliation":[{"name":"MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9684-5643","authenticated-orcid":false,"given":"Xuegong","family":"Zhang","sequence":"additional","affiliation":[{"name":"MOE Key Laboratory of Bioinformatics and Bioinformatics Division of BNRIST, Department of Automation, Tsinghua University , Beijing, 100084,","place":["China"]},{"name":"Center for Synthetic and Systems Biology, School of Life Sciences and School of Medicine, Tsinghua University , Beijing, 100084,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,7,7]]},"reference":[{"key":"2026071008251048600_btag227-B1","doi-asserted-by":"crossref","first-page":"D886","DOI":"10.1093\/nar\/gkae1142","article-title":"Cz cellxgene discover: a single-cell data platform for scalable exploration, analysis and modeling of aggregated data","volume":"53","author":"Abdulla","year":"2025","journal-title":"Nucleic Acids Res"},{"key":"2026071008251048600_btag227-B2","author":"Baek","year":"2024"},{"key":"2026071008251048600_btag227-B3","doi-asserted-by":"crossref","first-page":"btaf149","DOI":"10.1093\/bioinformatics\/btaf149","article-title":"Uhaf: a unified hierarchical annotation framework for cell type standardization and harmonization","volume":"41","author":"Bian","year":"2025","journal-title":"Bioinformatics"},{"key":"2026071008251048600_btag227-B4","doi-asserted-by":"crossref","first-page":"104318","DOI":"10.1016\/j.isci.2022.104318","article-title":"Heca: the cell-centric assembly of a cell atlas","volume":"25","author":"Chen","year":"2022","journal-title":"iScience"},{"key":"2026071008251048600_btag227-B5","doi-asserted-by":"crossref","first-page":"eabl5197","DOI":"10.1126\/science.abl5197","article-title":"Cross-tissue immune cell analysis reveals tissue-specific features in humans","volume":"376","author":"Dom\u00ednguez Conde","year":"2022","journal-title":"Science"},{"key":"2026071008251048600_btag227-B6","author":"Gottweis","year":"2025"},{"key":"2026071008251048600_btag227-B7","author":"Gu","year":"2024"},{"key":"2026071008251048600_btag227-B8","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1038\/s41576-023-00586-w","article-title":"Best practices for single-cell analysis across modalities","volume":"24","author":"Heumos","year":"2023","journal-title":"Nat Rev Genet"},{"key":"2026071008251048600_btag227-B9","doi-asserted-by":"crossref","DOI":"10.1101\/2025.05.30.656746","article-title":"Biomni: a general-purpose biomedical ai agent","author":"Huang","year":"2025"},{"key":"2026071008251048600_btag227-B10","author":"Jin","year":"2025"},{"key":"2026071008251048600_btag227-B11","author":"Laurent","year":"2024"},{"key":"2026071008251048600_btag227-B12","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1038\/s41587-024-02534-3","article-title":"A platform for the biomedical application of large language models","volume":"43","author":"Lobentanzer","year":"2025","journal-title":"Nat Biotechnol"},{"key":"2026071008251048600_btag227-B13","author":"Lu","year":"2024"},{"key":"2026071008251048600_btag227-B14","first-page":"2024","author":"Lu","year":"2024"},{"key":"2026071008251048600_btag227-B15","doi-asserted-by":"crossref","first-page":"e8746","DOI":"10.15252\/msb.20188746","article-title":"Current best practices in single-cell rna-seq analysis: a tutorial","volume":"15","author":"Luecken","year":"2019","journal-title":"Mol Syst Biol"},{"key":"2026071008251048600_btag227-B16","author":"Mitchener","year":"2025"},{"key":"2026071008251048600_btag227-B18","author":"Qi","year":"2024"},{"key":"2026071008251048600_btag227-B19","first-page":"28601","author":"Reddy","year":"2025"},{"key":"2026071008251048600_btag227-B20","author":"Schmidgall","year":"2025"},{"key":"2026071008251048600_btag227-B21","author":"Su","year":"2024"},{"key":"2026071008251048600_btag227-B22","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1038\/nprot.2017.149","article-title":"Exponential scaling of single-cell rna-seq in the past decade","volume":"13","author":"Svensson","year":"2018","journal-title":"Nat Protoc"},{"key":"2026071008251048600_btag227-B17","first-page":"2026","author":"Xu W"},{"key":"2026071008251048600_btag227-B23","doi-asserted-by":"crossref","DOI":"10.1038\/s41597-025-06426-2","article-title":"Heca v2. 0: an ai-ready ensemble cell atlas of single-cell rna and atac sequencing data","volume":"13","author":"Xi","year":"2025","journal-title":"Sci Data"},{"key":"2026071008251048600_btag227-B24","author":"Yamada","year":"2025"},{"key":"2026071008251048600_btag227-B25","author":"Zhang","year":"2024"},{"key":"2026071008251048600_btag227-B26","doi-asserted-by":"crossref","first-page":"14049","DOI":"10.1038\/ncomms14049","article-title":"Massively parallel digital transcriptional profiling of single cells","volume":"8","author":"Zheng","year":"2017","journal-title":"Nat Commun"},{"key":"2026071008251048600_btag227-B27","doi-asserted-by":"crossref","first-page":"2407094","DOI":"10.1002\/advs.202407094","article-title":"An ai agent for fully automated multi-omic analyses","volume":"11","author":"Zhou","year":"2024","journal-title":"Adv Sci"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/42\/Supplement_1\/btag227\/68709354\/btag227.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/42\/Supplement_1\/btag227\/68709354\/btag227.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T12:25:20Z","timestamp":1783686320000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btag227\/8726318"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,1]]},"references-count":27,"journal-issue":{"issue":"Supplement_1","published-print":{"date-parts":[[2026,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btag227","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,7]]},"published":{"date-parts":[[2026,7,1]]},"article-number":"btag227"}}