{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T19:42:43Z","timestamp":1776714163836,"version":"3.51.2"},"reference-count":63,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T00:00:00Z","timestamp":1765756800000},"content-version":"vor","delay-in-days":44,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Pioneer Hundred Talents Program of the Chinese Academy of Sciences and the Yunnan Talent Support Plan"},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["3250040463"],"award-info":[{"award-number":["3250040463"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Yunnan Basic Research Special Project\u2014Key Project","award":["202401AS070474"],"award-info":[{"award-number":["202401AS070474"]}]},{"name":"Yunnan Revitalization Talent Support Program Top team","award":["202505AT350003, 202405AS350022"],"award-info":[{"award-number":["202505AT350003, 202405AS350022"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models (LLMs) show significant potential for cell type annotation in single-cell RNA sequencing (scRNA-seq), but a systematic framework for their application and a comprehensive performance comparison are lacking. Here, we systematically benchmarked seven leading LLM models and three traditional bioinformatics tools across 34 diverse human and mouse datasets. We establish that using marker genes selected by statistical significance and ranked by log\u2082 fold change optimizes annotation accuracy. Our benchmark reveals that LLMs profoundly outperform traditional methods, particularly in resolving fine-grained cell subtypes. A top tier of models, including Kimi-k2, GPT-5, Claude-4.1, and Grok-4, consistently delivered the highest accuracy. To harness their collective strength, we developed an elite ensemble strategy that achieves state-of-the-art performance. We encapsulated these findings into DeepCellSeek, an open-source R package and interactive web platform, to provide a validated, high-performance solution. This work provides a practical roadmap for leveraging LLMs in single-cell research and paves the way for their evolution into powerful discovery engines.<\/jats:p>","DOI":"10.1093\/bib\/bbaf677","type":"journal-article","created":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T12:29:56Z","timestamp":1764592196000},"source":"Crossref","is-referenced-by-count":2,"title":["Benchmarking large language models for cell typing in single-cell RNA-Seq"],"prefix":"10.1093","volume":"26","author":[{"given":"Tianxiang","family":"Xiao","sequence":"first","affiliation":[{"name":"State Key Laboratory of Genetic Evolution & Animal Models, Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]},{"name":"College of Life Sciences, University of Chinese Academy of Sciences , No. 19(A) Yuquan Road, Shijingshan District, Beijing 100049 ,","place":["China"]},{"name":"Yunnan Key Laboratory of Biodiversity Information, Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]},{"name":"National Resource Center for Non-Human Primates, Kunming Primate Research Center, and National Research Facility for Phenotypic and Genetic Analysis of Model Animals (Primate Facility), Kunming Institute of Zoology, Chinese Academy of Sciences , Huahongdong, Wuhua District, Kunming, Yunnan 650107 ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dezhi","family":"Hua","sequence":"additional","affiliation":[{"name":"Biodiversity Data Center of Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7144-603X","authenticated-orcid":false,"given":"Yanan","family":"Wang","sequence":"additional","affiliation":[{"name":"Biodiversity Data Center of Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6044-6002","authenticated-orcid":false,"given":"Xuemei","family":"Lu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Genetic Evolution & Animal Models, Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]},{"name":"College of Life Sciences, University of Chinese Academy of Sciences , No. 19(A) Yuquan Road, Shijingshan District, Beijing 100049 ,","place":["China"]},{"name":"Yunnan Key Laboratory of Biodiversity Information, Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]},{"name":"National Resource Center for Non-Human Primates, Kunming Primate Research Center, and National Research Facility for Phenotypic and Genetic Analysis of Model Animals (Primate Facility), Kunming Institute of Zoology, Chinese Academy of Sciences , Huahongdong, Wuhua District, Kunming, Yunnan 650107 ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4167-4872","authenticated-orcid":false,"given":"Chao","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Genetic Evolution & Animal Models, Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]},{"name":"College of Life Sciences, University of Chinese Academy of Sciences , No. 19(A) Yuquan Road, Shijingshan District, Beijing 100049 ,","place":["China"]},{"name":"Yunnan Key Laboratory of Biodiversity Information, Kunming Institute of Zoology, Chinese Academy of Sciences , No. 17 Longxin Road, Panlong District, Kunming, Yunnan 650201 ,","place":["China"]},{"name":"National Resource Center for Non-Human Primates, Kunming Primate Research Center, and National Research Facility for Phenotypic and Genetic Analysis of Model Animals (Primate Facility), Kunming Institute of Zoology, Chinese Academy of Sciences , Huahongdong, Wuhua District, Kunming, Yunnan 650107 ,","place":["China"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2025,12,15]]},"reference":[{"key":"2025121511581754800_ref1","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1038\/nmeth.1315","article-title":"mRNA-Seq whole-transcriptome analysis of a single cell","volume":"6","author":"Tang","year":"2009","journal-title":"Nat Methods"},{"key":"2025121511581754800_ref2","doi-asserted-by":"crossref","first-page":"eaaq1723","DOI":"10.1126\/science.aaq1723","article-title":"Cell type atlas and lineage tree of a whole complex animal by single-cell transcriptomics","volume":"360","author":"Plass","year":"2018","journal-title":"Science"},{"key":"2025121511581754800_ref3","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/s13045-021-01105-2","article-title":"Applications of single-cell sequencing in cancer research: progress and perspectives","volume":"14","author":"Lei","year":"2021","journal-title":"J Hematol Oncol"},{"key":"2025121511581754800_ref4","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1038\/nri.2017.76","article-title":"Single-cell RNA sequencing to explore immune cell heterogeneity","volume":"18","author":"Papalexi","year":"2018","journal-title":"Nat Rev Immunol"},{"key":"2025121511581754800_ref5","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1186\/s13059-019-1795-z","article-title":"A comparison of automatic cell identification methods for single-cell RNA sequencing data","volume":"20","author":"Abdelaal","year":"2019","journal-title":"Genome Biol"},{"key":"2025121511581754800_ref6","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1038\/s41590-018-0276-y","article-title":"Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage","volume":"20","author":"Aran","year":"2019","journal-title":"Nat Immunol"},{"key":"2025121511581754800_ref7","doi-asserted-by":"publisher","first-page":"D870","DOI":"10.1093\/nar\/gkac947","article-title":"CellMarker 2.0: an updated database of manually curated cell markers in human\/mouse and web tools based on scRNA-seq data","volume":"51","author":"Hu","year":"2023","journal-title":"Nucleic Acids Res"},{"key":"2025121511581754800_ref8","doi-asserted-by":"crossref","first-page":"1246","DOI":"10.1038\/s41467-022-28803-w","article-title":"Fully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data","volume":"13","author":"Ianevski","year":"2022","journal-title":"Nat Commun"},{"key":"2025121511581754800_ref9","volume-title":"Nature","author":"Guo"},{"key":"2025121511581754800_ref10","author":"Anthropic"},{"key":"2025121511581754800_ref11","author":"Open AI"},{"key":"2025121511581754800_ref12","article-title":"Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic","author":"Comanici","year":"2025"},{"key":"2025121511581754800_ref13","author":"xAI"},{"key":"2025121511581754800_ref14","author":"ByteDance"},{"key":"2025121511581754800_ref15","author":"Moonshot AI"},{"key":"2025121511581754800_ref16","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025121511581754800_ref17","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025121511581754800_ref18","doi-asserted-by":"publisher","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Lin","year":"2023","journal-title":"Science"},{"key":"2025121511581754800_ref19","doi-asserted-by":"publisher","first-page":"1462","DOI":"10.1038\/s41592-024-02235-4","article-title":"Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis","volume":"21","author":"Hou","year":"2024","journal-title":"Nat Methods"},{"key":"2025121511581754800_ref20","doi-asserted-by":"crossref","DOI":"10.1101\/2023.07.13.548886","article-title":"Single cell portal: an interactive home for single-cell genomics","author":"Tarhan","year":"2023"},{"key":"2025121511581754800_ref21","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1038\/s41586-024-08216-z","article-title":"Spatially restricted immune and microbiota-driven adaptation of the gut","volume":"636","author":"Mayassi","year":"2024","journal-title":"Nature"},{"key":"2025121511581754800_ref22","doi-asserted-by":"crossref","first-page":"e3001532","DOI":"10.1371\/journal.pbio.3001532","article-title":"Colon stroma mediates an inflammation-driven fibroblastic response controlling matrix remodeling and healing","volume":"20","author":"Jasso","year":"2022","journal-title":"PLoS Biol"},{"key":"2025121511581754800_ref23","doi-asserted-by":"crossref","first-page":"4946","DOI":"10.1016\/j.cell.2024.07.002","article-title":"The choroid plexus synergizes with immune cells during neuroinflammation","volume":"187","author":"Xu","year":"2024","journal-title":"Cell"},{"key":"2025121511581754800_ref24","doi-asserted-by":"crossref","first-page":"e002809","DOI":"10.1136\/jitc-2021-002809","article-title":"Single-cell sequencing reveals antitumor characteristics of intratumoral immune cells in old mice","volume":"9","author":"Zhang","year":"2021","journal-title":"J Immunother Cancer"},{"key":"2025121511581754800_ref25","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1038\/s12276-021-00671-2","article-title":"Single-cell transcriptome analysis reveals cellular heterogeneity in the ascending aortas of normal and high-fat diet-fed mice","volume":"53","author":"Kan","year":"2021","journal-title":"Exp Mol Med"},{"key":"2025121511581754800_ref26","doi-asserted-by":"crossref","first-page":"7960","DOI":"10.1038\/s41467-022-35691-7","article-title":"Single-cell transcriptomic analysis identifies murine heart molecular features at embryonic and neonatal stages","volume":"13","author":"Feng","year":"2022","journal-title":"Nat Commun"},{"key":"2025121511581754800_ref27","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1038\/s41586-018-0590-4","article-title":"Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris: the Tabula Muris Consortium","volume":"562","author":"Schaum","year":"2018","journal-title":"Nature"},{"key":"2025121511581754800_ref28","doi-asserted-by":"crossref","DOI":"10.1172\/JCI141380","article-title":"Targeting a Braf\/Mapk pathway rescues podocyte lipid peroxidation in CoQ-deficiency kidney disease","volume":"131","author":"Sidhom","year":"2021","journal-title":"J Clin Invest"},{"key":"2025121511581754800_ref29","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1038\/s41586-018-0654-5","article-title":"Shared and distinct transcriptomic cell types across neocortical areas","volume":"563","author":"Tasic","year":"2018","journal-title":"Nature"},{"key":"2025121511581754800_ref30","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1038\/s41588-020-0636-z","article-title":"Lineage-dependent gene expression programs influence the immune landscape of colorectal cancer","volume":"52","author":"Lee","year":"2020","journal-title":"Nat Genet"},{"key":"2025121511581754800_ref31","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1016\/j.cels.2022.06.005","article-title":"Mapping hormone-regulated cell-cell interaction networks in the human breast at single-cell resolution","volume":"13","author":"Murrow","year":"2022","journal-title":"Cell Syst"},{"key":"2025121511581754800_ref32","doi-asserted-by":"crossref","first-page":"4383","DOI":"10.1038\/s41467-018-06318-7","article-title":"Single cell RNA sequencing of human liver reveals distinct intrahepatic macrophage populations","volume":"9","author":"MacParland","year":"2018","journal-title":"Nat Commun"},{"key":"2025121511581754800_ref33","article-title":"Single-cell transcriptome landscape of circulating CD4+ T cell populations in autoimmune diseases","volume":"4","author":"Yasumizu","year":"2024","journal-title":"Cell Genom"},{"key":"2025121511581754800_ref34","doi-asserted-by":"crossref","first-page":"e151621","DOI":"10.1172\/jci.insight.151621","article-title":"Combinatorial transcription factor profiles predict mature and functional human islet \u03b1 and \u03b2 cells","volume":"6","author":"Shrestha","year":"2021","journal-title":"JCI Insight"},{"key":"2025121511581754800_ref35","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1038\/s41586-020-2157-4","article-title":"Construction of a human cell landscape at single-cell level","volume":"581","author":"Han","year":"2020","journal-title":"Nature"},{"key":"2025121511581754800_ref36","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1038\/s41591-019-0468-5","article-title":"A cellular census of human lungs identifies novel cell states in health and in asthma","volume":"25","author":"Vieira Braga","year":"2019","journal-title":"Nat Med"},{"key":"2025121511581754800_ref37","doi-asserted-by":"crossref","first-page":"1888","DOI":"10.1016\/j.cell.2019.05.031","article-title":"Comprehensive integration of single-cell data","volume":"177","author":"Stuart","year":"2019","journal-title":"Cell"},{"key":"2025121511581754800_ref38","doi-asserted-by":"crossref","first-page":"eabl4896","DOI":"10.1126\/science.abl4896","article-title":"The Tabula Sapiens: a multiple-organ, single-cell transcriptomic atlas of humans","volume":"376","author":"Consortium* TTS","year":"2022","journal-title":"Science"},{"key":"2025121511581754800_ref39","doi-asserted-by":"publisher","first-page":"1334","DOI":"10.1038\/s41588-021-00911-1","article-title":"A single-cell and spatially resolved atlas of human breast cancers","volume":"53","author":"Wu","year":"2021","journal-title":"Nat Genet"},{"key":"2025121511581754800_ref40","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1016\/j.cell.2020.11.041","article-title":"Single-cell landscape of the ecosystem in early-relapse hepatocellular carcinoma","volume":"184","author":"Sun","year":"2021","journal-title":"Cell"},{"key":"2025121511581754800_ref41","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1038\/s41586-021-03569-1","article-title":"A molecular single-cell lung atlas of lethal COVID-19","volume":"595","author":"Melms","year":"2021","journal-title":"Nature"},{"key":"2025121511581754800_ref42","doi-asserted-by":"publisher","first-page":"2366","DOI":"10.1038\/s41593-024-01774-5","article-title":"Integrated multimodal cell atlas of Alzheimer\u2019s disease","volume":"27","author":"Gabitto","year":"2024","journal-title":"Nat Neurosci"},{"key":"2025121511581754800_ref43","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1093\/nar\/30.1.207","article-title":"Gene Expression Omnibus: NCBI gene expression and hybridization array data repository","volume":"30","author":"Edgar","year":"2002","journal-title":"Nucleic Acids Res"},{"key":"2025121511581754800_ref44","doi-asserted-by":"crossref","first-page":"D30","DOI":"10.1093\/nar\/gkae978","article-title":"Database resources of the National Genomics Data Center, China National Center for Bioinformation in 2025","volume":"53","author":"CNCB-NGDC Members and Partners","year":"2025","journal-title":"Nucleic Acids Res"},{"key":"2025121511581754800_ref45","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1038\/s41587-023-01767-y","article-title":"Dictionary learning for integrative, multimodal and scalable single-cell analysis","volume":"42","author":"Hao","year":"2024","journal-title":"Nat Biotechnol"},{"key":"2025121511581754800_ref46","article-title":"Gpt-4o system","author":"Hurst","year":"2024"},{"key":"2025121511581754800_ref47","author":"Anthropic"},{"key":"2025121511581754800_ref48","author":"Google DeepMind"},{"key":"2025121511581754800_ref49","author":"xAI"},{"key":"2025121511581754800_ref50","article-title":"The curious case of neural text degeneration","author":"Holtzman"},{"key":"2025121511581754800_ref51","first-page":"118","article-title":"A new ontology lookup service at EMBL-EBI","volume":"2","author":"Jupp","year":"2015","journal-title":"SWAT4LS"},{"key":"2025121511581754800_ref52","doi-asserted-by":"crossref","first-page":"632","DOI":"10.1186\/1471-2164-14-632","article-title":"An expression atlas of human primary cells: inference of gene function from coexpression networks","volume":"14","author":"Mabbott","year":"2013","journal-title":"BMC Genomics"},{"key":"2025121511581754800_ref53","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1101\/gr.240093.118","article-title":"Remodeling of epigenome and transcriptome landscapes with aging in mice reveals widespread induction of inflammatory responses","volume":"29","author":"Benayoun","year":"2019","journal-title":"Genome Res"},{"key":"2025121511581754800_ref54","article-title":"Large language model consensus substantially improves the cell type annotation accuracy for scRNA-seq data","author":"Yang","year":"2025","journal-title":"bioRxiv"},{"key":"2025121511581754800_ref55","article-title":"Celltypeagent: trustworthy cell type annotation with large language","author":"Chen","year":"2025"},{"key":"2025121511581754800_ref56","article-title":"scAgent: universal single-cell annotation via a LLM","author":"Mao","year":"2025"},{"key":"2025121511581754800_ref57","doi-asserted-by":"publisher","first-page":"9511","DOI":"10.1038\/s41467-025-64511-x","article-title":"Benchmarking cell type annotation by large language models with AnnDictionary","volume":"16","author":"Crowley","year":"2025","journal-title":"Nat Commun"},{"key":"2025121511581754800_ref58","article-title":"Scaling laws for neural language","author":"Kaplan","year":"2020"},{"key":"2025121511581754800_ref59","doi-asserted-by":"publisher","first-page":"833","DOI":"10.1056\/NEJMra2214964","article-title":"Considering biased data as informative artifacts in AI-assisted health care","volume":"389","author":"Ferryman","year":"2023","journal-title":"N Engl J Med"},{"key":"2025121511581754800_ref60","first-page":"1","volume-title":"International Workshop on Multiple Classifier Systems","author":"Dietterich","year":"2000"},{"key":"2025121511581754800_ref61","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2025121511581754800_ref62","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2025121511581754800_ref63","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inf Process Syst"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/6\/bbaf677\/65884586\/bbaf677.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/6\/bbaf677\/65884586\/bbaf677.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T16:58:30Z","timestamp":1765817910000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaf677\/8380197"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,1]]},"references-count":63,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaf677","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,11]]},"published":{"date-parts":[[2025,11,1]]},"article-number":"bbaf677"}}