{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,17]],"date-time":"2026-05-17T05:35:59Z","timestamp":1778996159197,"version":"3.51.4"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T00:00:00Z","timestamp":1723161600000},"content-version":"vor","delay-in-days":15,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Peak Disciplines"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,7,25]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Cell-type annotation is a critical step in single-cell data analysis. With the development of numerous cell annotation methods, it is necessary to evaluate these methods to help researchers use them effectively. Reference datasets are essential for evaluation, but currently, the cell labels of reference datasets mainly come from computational methods, which may have computational biases and may not reflect the actual cell-type outcomes. This study first constructed an experimentally labeled immune cell-subtype single-cell dataset of the same batch and systematically evaluated 18 cell annotation methods. We assessed those methods under five scenarios, including intra-dataset validation, immune cell-subtype validation, unsupervised clustering, inter-dataset annotation, and unknown cell-type prediction. Accuracy and ARI were evaluation metrics. The results showed that SVM, scBERT, and scDeepSort were the best-performing supervised methods. Seurat was the best-performing unsupervised clustering method, but it couldn't fully fit the actual cell-type distribution. Our results indicated that experimentally labeled immune cell-subtype datasets revealed the deficiencies of unsupervised clustering methods and provided new dataset support for supervised methods.<\/jats:p>","DOI":"10.1093\/bib\/bbae392","type":"journal-article","created":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T21:16:31Z","timestamp":1723238191000},"source":"Crossref","is-referenced-by-count":14,"title":["A comparison of scRNA-seq annotation methods based on experimentally labeled immune cell subtype dataset"],"prefix":"10.1093","volume":"25","author":[{"given":"Qiqing","family":"Fu","sequence":"first","affiliation":[{"name":"Institutes of Biomedical Sciences, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenyu","family":"Dong","sequence":"additional","affiliation":[{"name":"Institutes of Biomedical Sciences, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2120-952X","authenticated-orcid":false,"given":"Yunhe","family":"Liu","sequence":"additional","affiliation":[{"name":"Institutes of Biomedical Sciences, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoqiong","family":"Xia","sequence":"additional","affiliation":[{"name":"Institutes of Biomedical Sciences, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gang","family":"Liu","sequence":"additional","affiliation":[{"name":"Institutes of Biomedical Sciences, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fan","family":"Zhong","sequence":"additional","affiliation":[{"name":"Intelligent Medicine Institute, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9995-9080","authenticated-orcid":false,"given":"Lei","family":"Liu","sequence":"additional","affiliation":[{"name":"Intelligent Medicine Institute, Fudan University , 200032 Shanghai, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2024,8,9]]},"reference":[{"key":"2024091111053251400_ref1","doi-asserted-by":"crossref","first-page":"e109221","DOI":"10.15252\/embj.2021109221","article-title":"Deciphering functional tumor states at single-cell resolution","volume":"41","author":"Vegliante","year":"2022","journal-title":"EMBO J"},{"key":"2024091111053251400_ref2","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1038\/s43587-022-00198-9","article-title":"Multidimensional single-cell analysis of human peripheral blood reveals characteristic features of the immune system landscape in aging and frailty","volume":"2","author":"Luo","year":"2022","journal-title":"Nat Aging"},{"key":"2024091111053251400_ref3","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1016\/j.csbj.2021.01.015","article-title":"Automated methods for cell type annotation on scRNA-seq data","volume":"19","author":"Pasquini","year":"2021","journal-title":"Comput Struct Biotechnol J"},{"key":"2024091111053251400_ref4","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.gpb.2020.07.004","article-title":"Evaluation of cell type annotation R packages on single-cell RNA-seq data","volume":"19","author":"Huang","year":"2021","journal-title":"Genomics Proteomics Bioinformatics"},{"key":"2024091111053251400_ref5","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1093\/bib\/bbab567","article-title":"A comprehensive comparison of supervised and unsupervised methods for cell type identification in single-cell RNA-seq","volume":"23","author":"Sun","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024091111053251400_ref6","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1186\/s13059-019-1795-z","article-title":"A comparison of automatic cell identification methods for single-cell RNA sequencing data","volume":"20","author":"Abdelaal","year":"2019","journal-title":"Genome Biol"},{"key":"2024091111053251400_ref7","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1016\/j.cels.2016.09.002","article-title":"A single-cell transcriptome atlas of the human pancreas","volume":"3","author":"Muraro","year":"2016","journal-title":"Cell Syst"},{"key":"2024091111053251400_ref8","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1016\/j.cmet.2016.08.018","article-title":"RNA sequencing of single human islet cells reveals type 2 diabetes genes","volume":"24","author":"Xin","year":"2016","journal-title":"Cell Metab"},{"key":"2024091111053251400_ref9","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1038\/s41586-018-0590-4","article-title":"Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris","volume":"562","year":"2018","journal-title":"Nature"},{"key":"2024091111053251400_ref10","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.cels.2016.08.011","article-title":"A single-cell transcriptomic map of the human and mouse pancreas reveals inter- and intra-cell population structure","volume":"3","author":"Baron","year":"2016","journal-title":"Cell Syst"},{"key":"2024091111053251400_ref11","doi-asserted-by":"crossref","first-page":"14049","DOI":"10.1038\/ncomms14049","article-title":"Massively parallel digital transcriptional profiling of single cells","volume":"8","author":"Zheng","year":"2017","journal-title":"Nat Commun"},{"key":"2024091111053251400_ref12","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1038\/s41592-019-0425-8","article-title":"Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments","volume":"16","author":"Tian","year":"2019","journal-title":"Nat Methods"},{"key":"2024091111053251400_ref13","doi-asserted-by":"crossref","first-page":"852","DOI":"10.1038\/s42256-022-00534-z","article-title":"scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data","volume":"4","author":"Yang","year":"2022","journal-title":"Nat Mach Intell"},{"key":"2024091111053251400_ref14","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1038\/s41467-023-35923-4","article-title":"Transformer for one stop interpretable cell type annotation","volume":"14","author":"Chen","year":"2023","journal-title":"Nat Commun"},{"key":"2024091111053251400_ref15","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1038\/s41590-018-0276-y","article-title":"Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage","volume":"20","author":"Aran","year":"2019","journal-title":"Nat Immunol"},{"key":"2024091111053251400_ref16","doi-asserted-by":"crossref","first-page":"e95","DOI":"10.1093\/nar\/gkz543","article-title":"CHETAH: a selective, hierarchical cell type identification method for single-cell RNA sequencing","volume":"47","author":"Kanter","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2024091111053251400_ref17","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1038\/nmeth.4644","article-title":"scmap: projection of single-cell RNA-seq data across data sets","volume":"15","author":"Kiselev","year":"2018","journal-title":"Nat Methods"},{"key":"2024091111053251400_ref18","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1038\/s42256-020-00233-7","article-title":"Iterative transfer learning with neural network for clustering and cell type classification in single-cell RNA-seq analysis","volume":"2","author":"Hu","year":"2020","journal-title":"Nat Mach Intell"},{"key":"2024091111053251400_ref19","doi-asserted-by":"crossref","first-page":"3458","DOI":"10.1038\/s41467-020-17281-7","article-title":"Searching large-scale scRNA-seq databases via unbiased cell embedding with Cell BLAST","volume":"11","author":"Cao","year":"2020","journal-title":"Nat Commun"},{"key":"2024091111053251400_ref20","doi-asserted-by":"crossref","first-page":"e122","DOI":"10.1093\/nar\/gkab775","article-title":"scDeepSort: a pre-trained cell-type annotation method for single-cell transcriptomics using deep learning with a weighted graph neural network","volume":"49","author":"Shao","year":"2021","journal-title":"Nucleic Acids Res"},{"key":"2024091111053251400_ref21","doi-asserted-by":"crossref","first-page":"3573","DOI":"10.1016\/j.cell.2021.04.048","article-title":"Integrated analysis of multimodal single-cell data","volume":"184","author":"Hao","year":"2021","journal-title":"Cell"},{"key":"2024091111053251400_ref22","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1038\/s41586-019-0969-x","article-title":"The single-cell transcriptional landscape of mammalian organogenesis","volume":"566","author":"Cao","year":"2019","journal-title":"Nature"},{"key":"2024091111053251400_ref23","doi-asserted-by":"crossref","first-page":"979","DOI":"10.1038\/nmeth.4402","article-title":"Reversed graph embedding resolves complex single-cell trajectories","volume":"14","author":"Qiu","year":"2017","journal-title":"Nat Methods"},{"key":"2024091111053251400_ref24","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1101\/gr.254557.119","article-title":"SHARP: hyperfast and accurate processing of single-cell RNA-seq data via ensemble random projection","volume":"30","author":"Wan","year":"2020","journal-title":"Genome Res"},{"key":"2024091111053251400_ref25","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1038\/nmeth.4236","article-title":"SC3: consensus clustering of single-cell RNA-seq data","volume":"14","author":"Kiselev","year":"2017","journal-title":"Nat Methods"},{"key":"2024091111053251400_ref26","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btae020","article-title":"scMAE: a masked autoencoder for single-cell RNA-seq clustering","volume":"40","author":"Fang","year":"2024","journal-title":"Bioinformatics"},{"key":"2024091111053251400_ref27","doi-asserted-by":"crossref","DOI":"10.26508\/lsa.202302103","article-title":"scFseCluster: a feature selection-enhanced clustering for single-cell RNA-seq data","volume":"6","author":"Wang","year":"2023","journal-title":"Life Sci Alliance"},{"key":"2024091111053251400_ref28","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1038\/s42256-019-0037-0","article-title":"Clustering single-cell RNA-seq data with a model-based deep learning approach","volume":"1","author":"Tian","year":"2019","journal-title":"Nat Mach Intell"},{"key":"2024091111053251400_ref29","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1038\/s41587-020-0465-8","article-title":"Systematic comparison of single-cell and single-nucleus RNA-sequencing methods","volume":"38","author":"Ding","year":"2020","journal-title":"Nat Biotechnol"},{"key":"2024091111053251400_ref30","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1038\/s41591-020-0799-2","article-title":"Integrated single-cell analysis of multicellular immune dynamics during hyperacute HIV-1 infection","volume":"26","author":"Kazer","year":"2020","journal-title":"Nat Med"},{"key":"2024091111053251400_ref31","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1097\/00002030-199202000-00002","article-title":"Incidence of HIV infection in monocyte subpopulations characterized by CD4 and HLA-DR surface density","volume":"6","author":"Schnizlein-Bick","year":"1992","journal-title":"AIDS"},{"key":"2024091111053251400_ref32","doi-asserted-by":"crossref","first-page":"5556","DOI":"10.4049\/jimmunol.150.12.5556","article-title":"Neutrophil CD14: biochemical properties and role in the secretion of tumor necrosis factor-alpha in response to lipopolysaccharide","volume":"150","author":"Haziot","year":"1993","journal-title":"J Immunol"},{"key":"2024091111053251400_ref33","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1186\/s13059-021-02480-2","article-title":"Evaluation of some aspects in supervised cell type identification for single-cell RNA-seq: classifier, feature selection, and reference construction","volume":"22","author":"Ma","year":"2021","journal-title":"Genome Biol"},{"key":"2024091111053251400_ref34","doi-asserted-by":"crossref","first-page":"1287884","DOI":"10.3389\/fmed.2023.1287884","article-title":"Flow cytometry and cell sorting","volume":"10","author":"Telford","year":"2023","journal-title":"Front Med (Lausanne)"},{"key":"2024091111053251400_ref35","doi-asserted-by":"crossref","first-page":"181","DOI":"10.4155\/bio-2020-0267","article-title":"Flow cytometry: principles, applications and recent advances","volume":"13","author":"Manohar","year":"2021","journal-title":"Bioanalysis"},{"key":"2024091111053251400_ref36","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1080\/10826068.2012.719846","article-title":"Comparison of the sorting efficiency and influence on cell function between the sterile flow cytometry and immunomagnetic bead purification methods","volume":"43","author":"Li","year":"2013","journal-title":"Prep Biochem Biotechnol"},{"key":"2024091111053251400_ref37","doi-asserted-by":"crossref","first-page":"11","DOI":"10.3390\/magnetochemistry8010011","article-title":"Basic principles and recent advances in magnetic cell separation","volume":"8","author":"Frenea-Robin","year":"2022","journal-title":"Magnetochemistry"},{"key":"2024091111053251400_ref38","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1186\/s13059-020-1950-6","article-title":"Decontamination of ambient RNA in single-cell RNA-seq with DecontX","volume":"21","author":"Yang","year":"2020","journal-title":"Genome Biol"},{"key":"2024091111053251400_ref39","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1186\/s13059-019-1662-y","article-title":"EmptyDrops: distinguishing cells from empty droplets in droplet-based single-cell RNA sequencing data","volume":"20","author":"Lun","year":"2019","journal-title":"Genome Biol"},{"key":"2024091111053251400_ref40","doi-asserted-by":"crossref","DOI":"10.1093\/gigascience\/giaa151","article-title":"SoupX removes ambient RNA contamination from droplet-based single-cell RNA sequencing data","volume":"9","author":"Young","year":"2020","journal-title":"Gigascience"},{"key":"2024091111053251400_ref41","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1084\/jem.192.4.549","article-title":"Homeostasis-stimulated proliferation drives naive T cells to differentiate directly into memory T cells","volume":"192","author":"Cho","year":"2000","journal-title":"J Exp Med"},{"key":"2024091111053251400_ref42","doi-asserted-by":"crossref","first-page":"708","DOI":"10.1038\/44385","article-title":"Two subsets of memory T lymphocytes with distinct homing potentials and effector functions","volume":"401","author":"Sallusto","year":"1999","journal-title":"Nature"},{"key":"2024091111053251400_ref43","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1111\/jcpe.13399","article-title":"Peripheral memory T-cell profile is modified in patients undergoing periodontal management","volume":"48","author":"Medara","year":"2021","journal-title":"J Clin Periodontol"},{"key":"2024091111053251400_ref44","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1093\/intimm\/dxh218","article-title":"CD27(+) (memory) B cell decrease and apoptosis-resistant CD27(\u2212) (naive) B cell increase in aged humans: implications for age-related peripheral B cell developmental disturbances","volume":"17","author":"Chong","year":"2005","journal-title":"Int Immunol"},{"key":"2024091111053251400_ref45","doi-asserted-by":"crossref","first-page":"1169","DOI":"10.1038\/s41467-020-14976-9","article-title":"Embracing the dropouts in single-cell RNA-seq analysis","volume":"11","author":"Qiu","year":"2020","journal-title":"Nat Commun"},{"key":"2024091111053251400_ref46","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.imlet.2010.10.009","article-title":"Expression profiles of novel cell surface molecules on B-cell subsets and plasma cells as analyzed by flow cytometry","volume":"134","author":"Llin\u00e0s","year":"2011","journal-title":"Immunol Lett"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/5\/bbae392\/59069065\/bbae392.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/5\/bbae392\/59069065\/bbae392.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,11]],"date-time":"2024-09-11T11:22:58Z","timestamp":1726053778000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbae392\/7730135"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":46,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,7,25]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbae392","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,7,25]]},"article-number":"bbae392"}}