{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T02:29:28Z","timestamp":1785464968167,"version":"3.56.0"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2019,11,1]],"date-time":"2019-11-01T00:00:00Z","timestamp":1572566400000},"content-version":"vor","delay-in-days":9,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["81830053"],"award-info":[{"award-number":["81830053"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Shanghai Pujiang Program","award":["18PJ1432300"],"award-info":[{"award-number":["18PJ1432300"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,9,25]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Single-cell RNA sequencing (scRNA-seq) has been rapidly developing and widely applied in biological and medical research. Identification of cell types in scRNA-seq data sets is an essential step before in-depth investigations of their functional and pathological roles. However, the conventional workflow based on clustering and marker genes is not scalable for an increasingly large number of scRNA-seq data sets due to complicated procedures and manual annotation. Therefore, a number of tools have been developed recently to predict cell types in new data sets using reference data sets. These methods have not been generally adapted due to a lack of tool benchmarking and user guidance. In this article, we performed a comprehensive and impartial evaluation of nine classification software tools specifically designed for scRNA-seq data sets. Results showed that Seurat based on random forest, SingleR based on correlation analysis and CaSTLe based on XGBoost performed better than others. A simple ensemble voting of all tools can improve the predictive accuracy. Under nonideal situations, such as small-sized and class-imbalanced reference data sets, tools based on cluster-level similarities have superior performance. However, even with the function of assigning \u2018unassigned\u2019 labels, it is still challenging to catch novel cell types by solely using any of the single-cell classifiers. This article provides a guideline for researchers to select and apply suitable classification tools in their analysis workflows and sheds some lights on potential direction of future improvement on classification tools.<\/jats:p>","DOI":"10.1093\/bib\/bbz096","type":"journal-article","created":{"date-parts":[[2019,7,10]],"date-time":"2019-07-10T11:09:36Z","timestamp":1562756976000},"page":"1581-1595","source":"Crossref","is-referenced-by-count":95,"title":["Evaluation of single-cell classifiers for single-cell RNA sequencing data sets"],"prefix":"10.1093","volume":"21","author":[{"given":"Xinlei","family":"Zhao","sequence":"first","affiliation":[{"name":"State Key Laboratory of Bioelectronics, Biomedical Engineering School, Southeast University, Nanjing 210096, China"},{"name":"Singleron Biotechnologies, Nanjing 211800, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuang","family":"Wu","sequence":"additional","affiliation":[{"name":"Singleron Biotechnologies, Nanjing 211800, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nan","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Bioelectronics, Biomedical Engineering School, Southeast University, Nanjing 210096, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiao","family":"Sun","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Bioelectronics, Biomedical Engineering School, Southeast University, Nanjing 210096, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jue","family":"Fan","sequence":"additional","affiliation":[{"name":"Singleron Biotechnologies, Nanjing 211800, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,10,23]]},"reference":[{"key":"2021031109194454500_ref1","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/S2213-2600(16)00020-5","article-title":"Albert Coons: harnessing the power of the antibody","volume":"4","author":"Arthur","year":"2016","journal-title":"Lancet Respir Med"},{"key":"2021031109194454500_ref2","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1126\/science.150.3698.910","article-title":"Electronic separation of biological cells by volume","volume":"150","author":"Fulwyler","year":"1965","journal-title":"Science"},{"key":"2021031109194454500_ref3","volume-title":"Flow-Through Chamber for Photometers to Measure and Count Particles in a Dispersion Medium","author":"Dittrich"},{"key":"2021031109194454500_ref4","doi-asserted-by":"crossref","first-page":"3028","DOI":"10.1093\/bioinformatics\/btp524","article-title":"CellClassifier: supervised learning of cellular phenotypes","volume":"25","author":"Ramo","year":"2009","journal-title":"Bioinformatics"},{"key":"2021031109194454500_ref5","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1038\/nmeth.4179","article-title":"Seq-Well: portable, low-cost RNA sequencing of single cells at high throughput","volume":"14","author":"Gierahn","year":"2017","journal-title":"Nat Methods"},{"key":"2021031109194454500_ref6","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1016\/j.cell.2015.04.044","article-title":"Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells","volume":"161","author":"Klein","year":"2015","journal-title":"Cell"},{"key":"2021031109194454500_ref7","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1016\/j.cell.2015.05.002","article-title":"Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets","volume":"161","author":"Macosko","year":"2015","journal-title":"Cell"},{"key":"2021031109194454500_ref8","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1038\/nmeth.2639","article-title":"Smart-seq2 for sensitive full-length transcriptome profiling in single cells","volume":"10","author":"Picelli","year":"2013","journal-title":"Nat Methods"},{"key":"2021031109194454500_ref9","doi-asserted-by":"crossref","first-page":"14049","DOI":"10.1038\/ncomms14049","article-title":"Massively parallel digital transcriptional profiling of single cells","volume":"8","author":"Zheng","year":"2017","journal-title":"Nat Commun"},{"key":"2021031109194454500_ref10","doi-asserted-by":"crossref","first-page":"1307","DOI":"10.1016\/j.cell.2018.05.012","article-title":"Mapping the mouse cell atlas by Microwell-seq","volume":"173","author":"Han","year":"2018","journal-title":"Cell"},{"key":"2021031109194454500_ref11","doi-asserted-by":"crossref","first-page":"1258367","DOI":"10.1126\/science.1258367","article-title":"Expression profiling. Combinatorial labeling of single cells for gene expression cytometry","volume":"347","author":"Fan","year":"2015","journal-title":"Science"},{"key":"2021031109194454500_ref12","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1038\/nature11247","article-title":"An integrated encyclopedia of DNA elements in the human genome","volume":"489","author":"ENCODE Project Consortium","year":"2012","journal-title":"Nature"},{"key":"2021031109194454500_ref13","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1038\/s41586-018-0590-4","article-title":"Single-cell transcriptomics of 20 mouse organs creates a Tabula Muris","volume":"562","author":"Tabula Muris","year":"2018","journal-title":"Nature"},{"key":"2021031109194454500_ref14","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.27041","article-title":"The human cell atlas","volume":"6","author":"Regev","year":"2017","journal-title":"Elife"},{"key":"2021031109194454500_ref15","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1038\/s41586-019-0903-2","article-title":"Altered human oligodendrocyte heterogeneity in multiple sclerosis","volume":"566","author":"Jakel","year":"2019","journal-title":"Nature"},{"key":"2021031109194454500_ref16","doi-asserted-by":"crossref","first-page":"1867","DOI":"10.1038\/s41591-018-0269-2","article-title":"Single cell dissection of plasma cell heterogeneity in symptomatic and asymptomatic myeloma","volume":"24","author":"Ledergor","year":"2018","journal-title":"Nat Med"},{"key":"2021031109194454500_ref17","doi-asserted-by":"crossref","first-page":"984","DOI":"10.1016\/j.cell.2018.09.006","article-title":"A cancer cell program promotes T cell exclusion and resistance to checkpoint blockade","volume":"175","author":"Jerby-Arnon","year":"2018","journal-title":"Cell"},{"key":"2021031109194454500_ref18","doi-asserted-by":"crossref","first-page":"e1006245","DOI":"10.1371\/journal.pcbi.1006245","article-title":"Exploring the single-cell RNA-seq analysis landscape with the scRNA-tools database","volume":"14","author":"Zappia","year":"2018","journal-title":"PLoS Comput Biol"},{"key":"2021031109194454500_ref19","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1038\/nmeth.4644","article-title":"scmap: projection of single-cell RNA-seq data across data sets","volume":"15","author":"Kiselev","year":"2018","journal-title":"Nat Methods"},{"key":"2021031109194454500_ref20","first-page":"369538","article-title":"scPred: single cell prediction using singular value decomposition and machine learning classification","author":"Alquicira-Hernandez","year":"2018","journal-title":"bioRxiv"},{"key":"2021031109194454500_ref21","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1038\/nbt.3192","article-title":"Spatial reconstruction of single-cell gene expression data","volume":"33","author":"Satija","year":"2015","journal-title":"Nat Biotechnol"},{"key":"2021031109194454500_ref22","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1186\/s12859-019-2599-6","article-title":"Comparative analysis of differential gene expression analysis tools for single-cell RNA sequencing data","volume":"20","author":"Wang","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2021031109194454500_ref23","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1038\/nmeth.4612","article-title":"Bias, robustness and scalability in single-cell differential expression analysis","volume":"15","author":"Soneson","year":"2018","journal-title":"Nat Methods"},{"key":"2021031109194454500_ref24","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.12688\/f1000research.15666.2","article-title":"A systematic performance evaluation of clustering methods for single-cell RNA-seq data","volume":"7","author":"Duo","year":"2018","journal-title":"F1000Res"},{"key":"2021031109194454500_ref25","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.12688\/f1000research.15809.1","article-title":"Comparison of clustering tools in R for medium-sized 10x Genomics single-cell RNA-sequencing data","volume":"7","author":"Freytag","year":"2018","journal-title":"F1000Res"},{"key":"2021031109194454500_ref26","first-page":"276907","article-title":"A comparison of single-cell trajectory inference methods: towards more accurate and robust tools","author":"Saelens","year":"2018","journal-title":"bioRxiv"},{"key":"2021031109194454500_ref27","article-title":"Comparison of computational methods for imputing single-cell RNA-sequencing data","author":"Zhang","year":"2018","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2021031109194454500_ref28","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1186\/s13059-019-1639-x","article-title":"CellFishing.jl: an ultrafast and scalable cell search method for single-cell RNA sequencing","volume":"20","author":"Sato","year":"2019","journal-title":"Genome Biol"},{"key":"2021031109194454500_ref29","first-page":"e16","article-title":"scFTD-seq: freeze\u2013thaw lysis based, portable approach toward highly distributed single-cell 3\u2032 mRNA profiling","volume":"47","author":"Zhang","year":"2018","journal-title":"Nucleic Acids Res"},{"key":"2021031109194454500_ref30","first-page":"2579","article-title":"Visualizing Data using t-SNE","volume-title":"Journal of Machine Learning Research","author":"van der Maaten","year":"2008"},{"key":"2021031109194454500_ref31","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1038\/s41590-018-0276-y","article-title":"Reference-based analysis of lung single-cell sequencing reveals a transitional profibrotic macrophage","volume":"20","author":"Aran","year":"2019","journal-title":"Nat Immunol"},{"key":"2021031109194454500_ref32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v028.i05","article-title":"Building predictive models in R using the caret package","volume":"28","author":"Kuhn","year":"2008","journal-title":"J Stat Softw"},{"key":"2021031109194454500_ref33","volume-title":"Classification and Regression by RandomForest","author":"Liaw","year":"2001"},{"key":"2021031109194454500_ref34","doi-asserted-by":"crossref","first-page":"e0205499","DOI":"10.1371\/journal.pone.0205499","article-title":"CaSTLe\u2014classification of single cells by transfer learning: harnessing the power of publicly available single cell RNA sequencing experiments to annotate new experiments","volume":"13","author":"Lieberman","year":"2018","journal-title":"PLoS One"},{"key":"2021031109194454500_ref35","first-page":"470203","article-title":"scID: identification of equivalent transcriptional cell populations across single cell RNA-seq data using discriminant analysis","author":"Boufea","year":"2019","journal-title":"bioRxiv"},{"key":"2021031109194454500_ref36","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1038\/nature19348","article-title":"Single-cell analysis of mixed-lineage states leading to a binary cell fate choice","volume":"537","author":"Olsson","year":"2016","journal-title":"Nature"},{"key":"2021031109194454500_ref37","doi-asserted-by":"crossref","DOI":"10.1038\/s41467-018-03282-0","article-title":"Characterizing the replicability of cell types defined by single cell RNA-sequencing data using MetaNeighbor","volume":"9","author":"Crow","year":"2018","journal-title":"Nat Commun"},{"key":"2021031109194454500_ref38","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1126\/science.aan3235","article-title":"The Drosophila embryo at single-cell transcriptome resolution","volume":"358","author":"Karaiskos","year":"2017","journal-title":"Science"},{"key":"2021031109194454500_ref39","article-title":"Moana: a robust and scalable cell type classification framework for single-cell RNA-Seq data","author":"Wagner","year":"2018","journal-title":"bioRxiv"},{"key":"2021031109194454500_ref40","first-page":"2229","article-title":"Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation","volume-title":"J Mach Learn Technol","author":"Powers","year":"2011"},{"key":"2021031109194454500_ref41","doi-asserted-by":"crossref","first-page":"e41882","DOI":"10.1371\/journal.pone.0041882","article-title":"A comparison of MCC and CEN error measures in multi-class prediction","volume":"7","author":"Jurman","year":"2012","journal-title":"PLoS One"},{"key":"2021031109194454500_ref42","doi-asserted-by":"crossref","first-page":"e0177678","DOI":"10.1371\/journal.pone.0177678","article-title":"Optimal classifier for imbalanced data using Matthews correlation coefficient metric","volume":"12","author":"Boughorbel","year":"2017","journal-title":"PLoS One"},{"key":"2021031109194454500_ref43","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1038\/nbt.4096","article-title":"Integrating single-cell transcriptomic data across different conditions, technologies, and species","volume":"36","author":"Butler","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2021031109194454500_ref44","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1038\/nbt.4091","article-title":"Batch effects in single-cell RNA-sequencing data are corrected by matching mutual nearest neighbors","volume":"36","author":"Haghverdi","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2021031109194454500_ref45","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1038\/s41587-019-0113-3","article-title":"Efficient integration of heterogeneous single-cell transcriptomes using Scanorama","volume":"37","author":"Hie","year":"2019","journal-title":"Nat Biotechnol"},{"key":"2021031109194454500_ref46","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1038\/s41586-019-1127-1","article-title":"The emergent landscape of the mouse gut endoderm at single-cell resolution","volume":"569","author":"Nowotschin","year":"2019","journal-title":"Nature"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/21\/5\/1581\/36543433\/bbz096.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/21\/5\/1581\/36543433\/bbz096.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T11:29:44Z","timestamp":1615462184000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/21\/5\/1581\/5593804"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,23]]},"references-count":46,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2019,10,23]]},"published-print":{"date-parts":[[2020,9,25]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbz096","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,9]]},"published":{"date-parts":[[2019,10,23]]}}}