{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T01:14:43Z","timestamp":1784337283064,"version":"3.55.0"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T00:00:00Z","timestamp":1641859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61773401"],"award-info":[{"award-number":["61773401"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01GM122083"],"award-info":[{"award-number":["R01GM122083"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,3,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The cell type identification is among the most important tasks in single-cell RNA-sequencing (scRNA-seq) analysis. Many in silico methods have been developed and can be roughly categorized as either supervised or unsupervised. In this study, we investigated the performances of 8 supervised and 10 unsupervised cell type identification methods using 14 public scRNA-seq datasets of different tissues, sequencing protocols and species. We investigated the impacts of a number of factors, including total amount of cells, number of cell types, sequencing depth, batch effects, reference bias, cell population imbalance, unknown\/novel cell type, and computational efficiency and scalability. Instead of merely comparing individual methods, we focused on factors\u2019 impacts on the general category of supervised and unsupervised methods. We found that in most scenarios, the supervised methods outperformed the unsupervised methods, except for the identification of unknown cell types. This is particularly true when the supervised methods use a reference dataset with high informational sufficiency, low complexity and high similarity to the query dataset. However, such outperformance could be undermined by some undesired dataset properties investigated in this study, which lead to uninformative and biased reference datasets. In these scenarios, unsupervised methods could be comparable to supervised methods. Our study not only explained the cell typing methods\u2019 behaviors under different experimental settings but also provided a general guideline for the choice of method according to the scientific goal and dataset properties. Finally, our evaluation workflow is implemented as a modularized R pipeline that allows future evaluation of new methods.<\/jats:p><jats:p>Availability: All the source codes are available at https:\/\/github.com\/xsun28\/scRNAIdent.<\/jats:p>","DOI":"10.1093\/bib\/bbab567","type":"journal-article","created":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T20:13:10Z","timestamp":1639426390000},"source":"Crossref","is-referenced-by-count":56,"title":["A comprehensive comparison of supervised and unsupervised methods for cell type identification in single-cell RNA-seq"],"prefix":"10.1093","volume":"23","author":[{"given":"Xiaobo","family":"Sun","sequence":"first","affiliation":[{"name":"Department of Statistics, School of Statistics and Mathematics, Zhongnan University of Economics and Law, Wuhan, Hubei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaochu","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Rollins School of Public Health, Emory University, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8359-0533","authenticated-orcid":false,"given":"Ziyi","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, Texas, U.S"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1269-7354","authenticated-orcid":false,"given":"Hao","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Rollins School of Public Health, Emory University, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,1,11]]},"reference":[{"key":"2022031506265496500_ref1","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1038\/nmeth.1226","article-title":"Mapping and quantifying mammalian transcriptomes by RNA-Seq","volume":"5","author":"Mortazavi","year":"2008","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref2","doi-asserted-by":"crossref","first-page":"1581","DOI":"10.1093\/bib\/bbz096","article-title":"Evaluation of single-cell classifiers for single-cell RNA sequencing data sets","volume":"21","author":"Zhao","year":"2020","journal-title":"Brief Bioinform"},{"key":"2022031506265496500_ref3","doi-asserted-by":"crossref","first-page":"e1006245","DOI":"10.1371\/journal.pcbi.1006245","article-title":"Exploring the single-cell RNA-seq analysis landscape with the scRNA-tools database","volume":"14","author":"Zappia","year":"2018","journal-title":"PLoS Comput Biol"},{"key":"2022031506265496500_ref4","doi-asserted-by":"crossref","first-page":"296","DOI":"10.12688\/f1000research.18490.1","article-title":"Evaluation of methods to assign cell type labels to cell clusters from single-cell RNA-sequencing data","volume":"8","author":"Diaz-Mejia","year":"2019","journal-title":"F1000Research"},{"key":"2022031506265496500_ref5","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1038\/nmeth.3337","article-title":"Robust enumeration of cell subsets from tissue expression profiles","volume":"12","author":"Newman","year":"2015","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-018-03282-0","article-title":"Characterizing the replicability of cell types defined by single cell RNA-sequencing data using MetaNeighbor","volume":"9","author":"Crow","year":"2018","journal-title":"Nat Commun"},{"key":"2022031506265496500_ref7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13059-019-1795-z","article-title":"A comparison of automatic cell identification methods for single-cell RNA sequencing data","volume":"20","author":"Abdelaal","year":"2019","journal-title":"Genome Biol"},{"key":"2022031506265496500_ref8","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.gpb.2020.07.004","article-title":"Evaluation of cell type annotation R packages on single-cell RNA-seq data","volume":"19","author":"Huang","year":"2021","journal-title":"Genomics Proteomics Bioinformatics"},{"key":"2022031506265496500_ref9","doi-asserted-by":"crossref","first-page":"D721","DOI":"10.1093\/nar\/gky900","article-title":"CellMarker: a manually curated resource of cell markers in human and mouse","volume":"47","author":"Zhang","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2022031506265496500_ref10","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.12688\/f1000research.15666.2","article-title":"A systematic performance evaluation of clustering methods for single-cell RNA-seq data","volume":"7","author":"Du\u00f2","year":"2018","journal-title":"F1000Research"},{"key":"2022031506265496500_ref11","doi-asserted-by":"crossref","first-page":"1209","DOI":"10.1093\/bib\/bbz063","article-title":"Machine learning and statistical methods for clustering single-cell RNA-sequencing data","volume":"21","author":"Petegrosso","year":"2020","journal-title":"Brief Bioinform"},{"key":"2022031506265496500_ref12","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1007\/s10791-008-9066-8","article-title":"A comparison of extrinsic clustering evaluation metrics based on formal constraints","volume":"12","author":"Amig\u00f3","year":"2009","journal-title":"Information retrieval"},{"key":"2022031506265496500_ref13","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1038\/s41592-019-0529-1","article-title":"Probabilistic cell-type assignment of single-cell RNA-seq for tumor microenvironment profiling","volume":"16","author":"Zhang","year":"2019","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref14","doi-asserted-by":"crossref","first-page":"1888","DOI":"10.1016\/j.cell.2019.05.031","article-title":"Comprehensive integration of single-cell data","volume":"177","author":"Stuart","year":"2019","journal-title":"Cell"},{"key":"2022031506265496500_ref15","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1038\/nmeth.4644","article-title":"scmap: projection of single-cell RNA-seq data across data sets","volume":"15","author":"Kiselev","year":"2018","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref16","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1038\/s41590-018-0276-y","article-title":"Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage","volume":"20","author":"Aran","year":"2019","journal-title":"Nat Immunol"},{"key":"2022031506265496500_ref17","article-title":"CHETAH: a selective, hierarchical cell type identification method for single-cell RNA sequencing","volume":"47","author":"Kanter","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2022031506265496500_ref18","doi-asserted-by":"crossref","first-page":"983","DOI":"10.1038\/s41592-019-0535-3","article-title":"Supervised classification enables rapid annotation of cell atlases","volume":"16","author":"Pliner","year":"2019","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref19","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.cels.2019.06.004","article-title":"SingleCellNet: a computational tool to classify single cell RNA-Seq data across platforms and across species","volume":"9","author":"Tan","year":"2019","journal-title":"Cell Syst"},{"key":"2022031506265496500_ref20","first-page":"379","article-title":"FateID infers cell fate bias in multipotent progenitors from single-cell RNA-seq data","author":"Herman","year":"2018"},{"key":"2022031506265496500_ref21","doi-asserted-by":"crossref","first-page":"1873","DOI":"10.1016\/j.cell.2019.05.006","article-title":"Single-cell multi-omic integration compares and contrasts features of brain cell identity","volume":"177","author":"Welch","year":"2019","journal-title":"Cell"},{"key":"2022031506265496500_ref22","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1038\/nmeth.4236","article-title":"SC3: consensus clustering of single-cell RNA-seq data","volume":"14","author":"Kiselev","year":"2017","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref23","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1038\/s41586-019-0969-x","article-title":"The single-cell transcriptional landscape of mammalian organogenesis","volume":"566","author":"Cao","year":"2019","journal-title":"Nature"},{"key":"2022031506265496500_ref24","doi-asserted-by":"crossref","first-page":"e117","DOI":"10.1093\/nar\/gkw430","article-title":"TSCAN: pseudo-time reconstruction and evaluation in single-cell RNA-seq analysis","volume":"44","author":"Ji","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2022031506265496500_ref25","first-page":"1","article-title":"pcaReduce: hierarchical clustering of single cell transcriptional profiles","volume":"17","author":"Yau","year":"2016","journal-title":"BMC bioinformatics"},{"key":"2022031506265496500_ref26","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1186\/s13059-017-1188-0","article-title":"CIDR: ultrafast and accurate clustering through imputation for single-cell RNA-seq data","volume":"18","author":"Lin","year":"2017","journal-title":"Genome Biol"},{"key":"2022031506265496500_ref27","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1093\/nar\/gkz959","article-title":"SAME-clustering: single-cell aggregated clustering via mixture model ensemble","volume":"48","author":"Huh","year":"2020","journal-title":"Nucleic Acids Res"},{"key":"2022031506265496500_ref28","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1101\/gr.254557.119","article-title":"SHARP: hyperfast and accurate processing of single-cell RNA-seq data via ensemble random projection","volume":"30","author":"Wan","year":"2020","journal-title":"Genome Res"},{"key":"2022031506265496500_ref29","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1038\/nbt.4091","article-title":"Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors","volume":"36","author":"Haghverdi","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2022031506265496500_ref30","doi-asserted-by":"crossref","first-page":"14049","DOI":"10.1038\/ncomms14049","article-title":"Massively parallel digital transcriptional profiling of single cells","volume":"8","author":"Zheng","year":"2017","journal-title":"Nat Commun"},{"key":"2022031506265496500_ref31","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/nbt.4042","article-title":"Multiplexed droplet single-cell RNA-sequencing using natural genetic variation","volume":"36","author":"Kang","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2022031506265496500_ref32","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1016\/j.cmet.2016.08.020","article-title":"Single-cell transcriptome profiling of human pancreatic islets in health and type 2 diabetes","volume":"24","author":"Segerstolpe","year":"2016","journal-title":"Cell Metab"},{"key":"2022031506265496500_ref33","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1016\/j.cels.2016.09.002","article-title":"A single-cell transcriptome atlas of the human pancreas","volume":"3","author":"Muraro","year":"2016","journal-title":"Cell Syst"},{"key":"2022031506265496500_ref34","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1016\/j.cmet.2016.08.018","article-title":"RNA sequencing of single human islet cells reveals type 2 diabetes genes","volume":"24","author":"Xin","year":"2016","journal-title":"Cell Metab"},{"key":"2022031506265496500_ref35","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1038\/s41586-019-1195-2","article-title":"Single-cell transcriptomic analysis of Alzheimer's disease","volume":"570","author":"Mathys","year":"2019","journal-title":"Nature"},{"key":"2022031506265496500_ref36","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1126\/science.aav8130","article-title":"Single-cell genomics identifies cell type-specific molecular changes in autism","volume":"364","author":"Velmeshev","year":"2019","journal-title":"Science"},{"key":"2022031506265496500_ref37","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1016\/j.cell.2016.09.027","article-title":"Molecular diversity of midbrain development in mouse, human, and stem cells","volume":"167","author":"La Manno","year":"2016","journal-title":"Cell"},{"key":"2022031506265496500_ref38","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1038\/s41586-020-2157-4","article-title":"Construction of a human cell landscape at single-cell level","volume":"581","author":"Han","year":"2020","journal-title":"Nature"},{"key":"2022031506265496500_ref39","doi-asserted-by":"crossref","first-page":"1179","DOI":"10.1093\/bioinformatics\/btw777","article-title":"Scater: pre-processing, quality control, normalization and visualization of single-cell RNA-seq data in R","volume":"33","author":"McCarthy","year":"2017","journal-title":"Bioinformatics"},{"key":"2022031506265496500_ref40","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1093\/biostatistics\/kxx053","article-title":"Missing data and technical variability in single-cell RNA-sequencing experiments","volume":"19","author":"Hicks","year":"2018","journal-title":"Biostatistics"},{"key":"2022031506265496500_ref41","doi-asserted-by":"crossref","first-page":"39921","DOI":"10.1038\/srep39921","article-title":"Batch effects and the effective design of single-cell gene expression studies","volume":"7","author":"Tung","year":"2017","journal-title":"Sci Rep"},{"key":"2022031506265496500_ref42","doi-asserted-by":"crossref","first-page":"lqaa078","DOI":"10.1093\/nargab\/lqaa078","article-title":"ComBat-seq: batch effect adjustment for RNA-seq count data","volume":"2","author":"Zhang","year":"2020","journal-title":"NAR Genom Bioinform"},{"key":"2022031506265496500_ref43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/bioinformatics\/btv544","article-title":"A non-negative matrix factorization method for detecting modules in heterogeneous omics multi-modal data","volume":"32","author":"Yang","year":"2016","journal-title":"Bioinformatics"},{"key":"2022031506265496500_ref44","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbab122","article-title":"SMNN: batch effect correction for single-cell RNA-seq data via supervised mutual nearest neighbor detection","volume":"22","author":"Yang","year":"2021","journal-title":"Brief Bioinform"},{"key":"2022031506265496500_ref45","doi-asserted-by":"crossref","first-page":"1289","DOI":"10.1038\/s41592-019-0619-0","article-title":"Fast, sensitive and accurate integration of single-cell data with harmony","volume":"16","author":"Korsunsky","year":"2019","journal-title":"Nat Methods"},{"key":"2022031506265496500_ref46","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1007\/s10898-013-0035-4","article-title":"Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework","volume":"58","author":"Kim","year":"2014","journal-title":"J Global Optimizat"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/2\/bbab567\/42805856\/bbab567.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/2\/bbab567\/42805856\/bbab567.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T01:38:39Z","timestamp":1699925919000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbab567\/6502554"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,11]]},"references-count":46,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,3,10]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbab567","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,3]]},"published":{"date-parts":[[2022,1,11]]},"article-number":"bbab567"}}