{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T16:40:36Z","timestamp":1775580036087,"version":"3.50.1"},"reference-count":43,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2024,1,3]],"date-time":"2024-01-03T00:00:00Z","timestamp":1704240000000},"content-version":"vor","delay-in-days":42,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82301909"],"award-info":[{"award-number":["82301909"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["82171664"],"award-info":[{"award-number":["82171664"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Science and Technology Project of Chongqing Education Committee","award":["KJZD-K202200408"],"award-info":[{"award-number":["KJZD-K202200408"]}]},{"name":"Science and Technology Project of Chongqing Education Committee","award":["KJQN202100421"],"award-info":[{"award-number":["KJQN202100421"]}]},{"name":"Natural Science Foundation of Chongqing Municipality of China","award":["CSTB2022NS"],"award-info":[{"award-number":["CSTB2022NS"]}]},{"name":"Natural Science Foundation of Chongqing Municipality of China","award":["CQ-LZX0062"],"award-info":[{"award-number":["CQ-LZX0062"]}]},{"name":"Basic Research and Frontiers Exploration Project of Science and Technology Committee of Yuzhong District, Chongqing","award":["20210119"],"award-info":[{"award-number":["20210119"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,11,22]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Cell clustering is typically the initial step in single-cell RNA sequencing (scRNA-seq) analyses. The performance of clustering considerably impacts the validity and reproducibility of cell identification. A variety of clustering algorithms have been developed for scRNA-seq data. These algorithms generate cell label sets that assign each cell to a cluster. However, different algorithms usually yield different label sets, which can introduce variations in cell-type identification based on the generated label sets. Currently, the performance of these algorithms has not been systematically evaluated in single-cell transcriptome studies. Herein, we performed a critical assessment of seven state-of-the-art clustering algorithms including four deep learning-based clustering algorithms and commonly used methods Seurat, Cosine-based Tanimoto similarity-refined graph for community detection using Leiden\u2019s algorithm (CosTaL) and Single-cell consensus clustering (SC3). We used diverse evaluation indices based on 10 different scRNA-seq benchmarks to systematically evaluate their clustering performance. Our results show that CosTaL, Seurat, Deep Embedding for Single-cell Clustering (DESC) and SC3 consistently outperformed Single-Cell Clustering Assessment Framework and scDeepCluster based on nine effectiveness scores. Notably, CosTaL and DESC demonstrated superior performance in clustering specific cell types. The performance of the single-cell Variational Inference tools varied across different datasets, suggesting its sensitivity to certain dataset characteristics. Notably, DESC exhibited promising results for cell subtype identification and capturing cellular heterogeneity. In addition, SC3 requires more memory and exhibits slower computation speed compared to other algorithms for the same dataset. In sum, this study provides useful guidance for selecting appropriate clustering methods in scRNA-seq data analysis.<\/jats:p>","DOI":"10.1093\/bib\/bbad497","type":"journal-article","created":{"date-parts":[[2024,1,3]],"date-time":"2024-01-03T09:20:25Z","timestamp":1704273625000},"source":"Crossref","is-referenced-by-count":8,"title":["A critical assessment of clustering algorithms to improve cell clustering and identification in single-cell transcriptome study"],"prefix":"10.1093","volume":"25","author":[{"given":"Xiao","family":"Liang","sequence":"first","affiliation":[{"name":"Department of Obstetrics and Gynecology, Women and Children\u2019s Hospital of Chongqing Medical University , Chongqing 401147, China"},{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lijie","family":"Cao","sequence":"additional","affiliation":[{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lidan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yangyun","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lijuan","family":"Fu","sequence":"additional","affiliation":[{"name":"Joint International Research Laboratory of Reproduction and Development of the Ministry of Education of China , School of Public Health, , Chongqing 400016, China"},{"name":"Chongqing Medical University , School of Public Health, , Chongqing 400016, China"},{"name":"Department of Pharmacology , Academician Workstation, , Changsha 410219, China"},{"name":"Changsha Medical University , Academician Workstation, , Changsha 410219, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaqin","family":"Tan","sequence":"additional","affiliation":[{"name":"The First Affiliated Hospital of Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Enxiang","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"},{"name":"Joint International Research Laboratory of Reproduction and Development of the Ministry of Education of China , School of Public Health, , Chongqing 400016, China"},{"name":"Chongqing Medical University , School of Public Health, , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yubin","family":"Ding","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology, Women and Children\u2019s Hospital of Chongqing Medical University , Chongqing 401147, China"},{"name":"Joint International Research Laboratory of Reproduction and Development of the Ministry of Education of China , School of Public Health, , Chongqing 400016, China"},{"name":"Chongqing Medical University , School of Public Health, , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2645-2032","authenticated-orcid":false,"given":"Jing","family":"Tang","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology, Women and Children\u2019s Hospital of Chongqing Medical University , Chongqing 401147, China"},{"name":"School of Basic Medicine, Chongqing Medical University , Chongqing 400016, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2024,1,2]]},"reference":[{"key":"2024011113572277500_ref1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41596-020-00409-w","article-title":"Tutorial: guidelines for the computational analysis of single-cell RNA sequencing data","volume":"16","author":"Andrews","year":"2021","journal-title":"Nat Protoc"},{"key":"2024011113572277500_ref2","doi-asserted-by":"crossref","first-page":"e694","DOI":"10.1002\/ctm2.694","article-title":"Single-cell RNA sequencing technologies and applications: a brief overview","volume":"12","author":"Jovic","year":"2022","journal-title":"Clin Transl Med"},{"key":"2024011113572277500_ref3","doi-asserted-by":"crossref","first-page":"1131","DOI":"10.1038\/nsmb.2660","article-title":"Single-cell RNA-Seq profiling of human preimplantation embryos and embryonic stem cells","volume":"20","author":"Yan","year":"2013","journal-title":"Nat Struct Mol Biol"},{"key":"2024011113572277500_ref4","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1016\/j.cell.2015.04.044","article-title":"Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells","volume":"161","author":"Klein","year":"2015","journal-title":"Cell"},{"key":"2024011113572277500_ref5","doi-asserted-by":"crossref","first-page":"496","DOI":"10.1038\/s41573-023-00688-4","article-title":"Applications of single-cell RNA sequencing in drug discovery and development","volume":"22","author":"Van de Sande","year":"2023","journal-title":"Nat Rev Drug Discov"},{"key":"2024011113572277500_ref6","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1038\/nbt.2859","article-title":"The dynamics and regulators of cell fate decisions are revealed by pseudotemporal ordering of single cells","volume":"32","author":"Trapnell","year":"2014","journal-title":"Nat Biotechnol"},{"key":"2024011113572277500_ref7","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1186\/s13059-022-02605-1","article-title":"IDEAS: individual level differential expression analysis for single-cell RNA-seq data","volume":"23","author":"Zhang","year":"2022","journal-title":"Genome Biol"},{"key":"2024011113572277500_ref8","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1038\/s41576-022-00449-w","article-title":"Single-cell atlases: shared and tissue-specific cell types across human organs","volume":"23","author":"Elmentaite","year":"2022","journal-title":"Nat Rev Genet"},{"key":"2024011113572277500_ref9","doi-asserted-by":"crossref","first-page":"1277","DOI":"10.1038\/s41591-018-0096-5","article-title":"Phenotype molding of stromal cells in the lung tumor microenvironment","volume":"24","author":"Lambrechts","year":"2018","journal-title":"Nat Med"},{"key":"2024011113572277500_ref10","doi-asserted-by":"crossref","first-page":"1731","DOI":"10.3389\/fimmu.2020.01731","article-title":"Redefining tumor-associated macrophage subpopulations and functions in the tumor microenvironment","volume":"11","author":"Wu","year":"2020","journal-title":"Front Immunol"},{"key":"2024011113572277500_ref11","doi-asserted-by":"crossref","first-page":"1383","DOI":"10.1016\/j.cell.2020.10.002","article-title":"Single-cell profiling of Ebola virus disease in vivo reveals viral and host dynamics","volume":"183","author":"Kotliar","year":"2020","journal-title":"Cell"},{"key":"2024011113572277500_ref12","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.1016\/j.cell.2020.04.035","article-title":"SARS-CoV-2 receptor ACE2 is an interferon-stimulated gene in human airway epithelial cells and is detected in specific cell subsets across tissues","volume":"181","author":"Ziegler","year":"2020","journal-title":"Cell"},{"key":"2024011113572277500_ref13","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1038\/s41569-020-0359-y","article-title":"Single-cell RNA sequencing in cardiovascular development, disease and medicine","volume":"17","author":"Paik","year":"2020","journal-title":"Nat Rev Cardiol"},{"key":"2024011113572277500_ref14","doi-asserted-by":"crossref","first-page":"100882","DOI":"10.1016\/j.isci.2020.100882","article-title":"scCATCH: automatic annotation on cell types of clusters from single-cell RNA sequencing data","volume":"23","author":"Shao","year":"2020","journal-title":"iScience"},{"key":"2024011113572277500_ref15","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1186\/s13059-022-02825-5","article-title":"Clustering Deviation Index (CDI): a robust and accurate internal measure for evaluating scRNA-seq data clustering","volume":"23","author":"Fang","year":"2022","journal-title":"Genome Biol"},{"key":"2024011113572277500_ref16","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1186\/s13059-019-1850-9","article-title":"A benchmark of batch-effect correction methods for single-cell RNA sequencing data","volume":"21","author":"Tran","year":"2020","journal-title":"Genome Biol"},{"key":"2024011113572277500_ref17","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1186\/s13059-020-02103-2","article-title":"Bayesian model selection reveals biological origins of zero inflation in single-cell transcriptomics","volume":"21","author":"Choi","year":"2020","journal-title":"Genome Biol"},{"key":"2024011113572277500_ref18","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1038\/s41576-019-0095-5","article-title":"Publisher correction: challenges in unsupervised clustering of single-cell RNA-seq data","volume":"20","author":"Kiselev","year":"2019","journal-title":"Nat Rev Genet"},{"key":"2024011113572277500_ref19","doi-asserted-by":"crossref","first-page":"2338","DOI":"10.1038\/s41467-020-15851-3","article-title":"Deep learning enables accurate clustering with batch effect removal in single-cell RNA-seq analysis","volume":"11","author":"Li","year":"2020","journal-title":"Nat Commun"},{"key":"2024011113572277500_ref20","doi-asserted-by":"crossref","first-page":"1053","DOI":"10.1038\/s41592-018-0229-2","article-title":"Deep generative modeling for single-cell transcriptomics","volume":"15","author":"Lopez","year":"2018","journal-title":"Nat Methods"},{"key":"2024011113572277500_ref21","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1038\/s42256-019-0037-0","article-title":"Clustering single-cell RNA-seq data with a model-based deep learning approach","volume":"1","author":"Tian","year":"2019","journal-title":"Nat Mach Intell"},{"key":"2024011113572277500_ref22","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1038\/s41592-020-0825-9","article-title":"Putative cell type discovery from single-cell gene expression data","volume":"17","author":"Miao","year":"2020","journal-title":"Nat Methods"},{"key":"2024011113572277500_ref23","doi-asserted-by":"crossref","first-page":"814","DOI":"10.1016\/j.gpb.2022.11.011","article-title":"Application of deep learning on single-cell RNA sequencing data analysis: a review","volume":"20","author":"Brendel","year":"2022","journal-title":"Genom Proteom Bioinform"},{"key":"2024011113572277500_ref24","doi-asserted-by":"crossref","first-page":"bbab090","DOI":"10.1093\/bib\/bbab090","article-title":"Deep embedded clustering with multiple objectives on scRNA-seq data","volume":"22","author":"Li","year":"2021","journal-title":"Brief Bioinform"},{"key":"2024011113572277500_ref25","doi-asserted-by":"crossref","first-page":"4497","DOI":"10.1093\/bioinformatics\/btac575","article-title":"CLNN-loop: a deep learning model to predict CTCF-mediated chromatin loops in the different cell lines and CTCF-binding sites (CBS) pair types","volume":"38","author":"Zhang","year":"2022","journal-title":"Bioinformatics"},{"key":"2024011113572277500_ref26","doi-asserted-by":"crossref","first-page":"4559","DOI":"10.1109\/JBHI.2023.3292299","article-title":"IChrom-deep: an attention-based deep learning model for identifying chromatin interactions","volume":"27","author":"Zhang","year":"2023","journal-title":"IEEE J Biomed Health Inform"},{"key":"2024011113572277500_ref27","doi-asserted-by":"crossref","first-page":"10278","DOI":"10.1093\/nar\/gkac824","article-title":"iPro-WAEL: a comprehensive and robust framework for identifying promoters in multiple species","volume":"50","author":"Zhang","year":"2022","journal-title":"Nucleic Acids Res"},{"key":"2024011113572277500_ref28","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1038\/nmeth.4236","article-title":"SC3: consensus clustering of single-cell RNA-seq data","volume":"14","author":"Kiselev","year":"2017","journal-title":"Nat Methods"},{"key":"2024011113572277500_ref29","doi-asserted-by":"crossref","first-page":"2778","DOI":"10.1093\/bioinformatics\/btaa042","article-title":"PARC: ultrafast and accurate clustering of phenotypic data of millions of single cells","volume":"36","author":"Stassen","year":"2020","journal-title":"Bioinformatics"},{"key":"2024011113572277500_ref30","doi-asserted-by":"crossref","first-page":"bbad157","DOI":"10.1093\/bib\/bbad157","article-title":"CosTaL: an accurate and scalable graph-based clustering algorithm for high-dimensional single-cell data analysis","volume":"24","author":"Li","year":"2023","journal-title":"Brief Bioinform"},{"key":"2024011113572277500_ref31","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1093\/bfgp\/elad004","article-title":"SCMcluster: a high-precision cell clustering algorithm integrating marker gene set with single-cell RNA sequencing data","volume":"22","author":"Wu","year":"2023","journal-title":"Brief Funct Genomics"},{"key":"2024011113572277500_ref32","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1038\/s41587-020-0469-4","article-title":"Benchmarking single-cell RNA-sequencing protocols for cell atlas projects","volume":"38","author":"Mereu","year":"2020","journal-title":"Nat Biotechnol"},{"key":"2024011113572277500_ref33","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1186\/s13073-017-0467-4","article-title":"A practical guide to single-cell RNA-sequencing for biomedical research and clinical applications","volume":"9","author":"Haque","year":"2017","journal-title":"Genome Med"},{"key":"2024011113572277500_ref34","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.mam.2017.07.002","article-title":"Identifying cell populations with scRNASeq","volume":"59","author":"Andrews","year":"2018","journal-title":"Mol Aspects Med"},{"key":"2024011113572277500_ref35","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1038\/nbt.4314","article-title":"Dimensionality reduction for visualizing single-cell data using UMAP","volume":"37","author":"Becht","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2024011113572277500_ref36","doi-asserted-by":"crossref","first-page":"1141","DOI":"10.12688\/f1000research.15666.2","article-title":"A systematic performance evaluation of clustering methods for single-cell RNA-seq data","volume":"7","author":"Duo","year":"2018","journal-title":"F1000Res"},{"key":"2024011113572277500_ref37","doi-asserted-by":"crossref","first-page":"7380","DOI":"10.1093\/nar\/gkr462","article-title":"An intuitive graphical visualization technique for the interrogation of transcriptome data","volume":"39","author":"Bushati","year":"2011","journal-title":"Nucleic Acids Res"},{"key":"2024011113572277500_ref38","doi-asserted-by":"crossref","first-page":"e0261531","DOI":"10.1371\/journal.pone.0261531","article-title":"DeLUCS: deep learning for unsupervised clustering of DNA sequences","volume":"17","author":"Millan Arias","year":"2022","journal-title":"PloS One"},{"key":"2024011113572277500_ref39","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1109\/TNN.2005.845141","article-title":"Survey of clustering algorithms","volume":"16","author":"Xu","year":"2005","journal-title":"IEEE Trans Neural Netw"},{"key":"2024011113572277500_ref40","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1093\/bioinformatics\/btx490","article-title":"DIMM-SC: a Dirichlet mixture model for clustering droplet-based single cell transcriptomic data","volume":"34","author":"Sun","year":"2018","journal-title":"Bioinformatics"},{"key":"2024011113572277500_ref41","doi-asserted-by":"crossref","first-page":"bbac172","DOI":"10.1093\/bib\/bbac172","article-title":"A parameter-free deep embedded clustering method for single-cell RNA-seq data","volume":"23","author":"Zeng","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024011113572277500_ref42","doi-asserted-by":"crossref","first-page":"811043","DOI":"10.3389\/fgene.2021.811043","article-title":"Dissecting cellular heterogeneity based on network denoising of scRNA-seq using local scaling self-diffusion","volume":"12","author":"Duan","year":"2021","journal-title":"Front Genet"},{"key":"2024011113572277500_ref43","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1038\/s41422-018-0066-y","article-title":"Single-cell RNA-seq reveals the diversity of trophoblast subtypes and patterns of differentiation in the human placenta","volume":"28","author":"Liu","year":"2018","journal-title":"Cell Res"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/1\/bbad497\/55381300\/bbad497.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/1\/bbad497\/55381300\/bbad497.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,11]],"date-time":"2024-01-11T14:59:06Z","timestamp":1704985146000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbad497\/7505072"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,22]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,11,22]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbad497","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,1,1]]},"published":{"date-parts":[[2023,11,22]]},"article-number":"bbad497"}}