{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T23:14:44Z","timestamp":1787008484273,"version":"build-2736575974"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2018,8,28]],"date-time":"2018-08-28T00:00:00Z","timestamp":1535414400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","award":["1902\/12"],"award-info":[{"award-number":["1902\/12"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","award":["1634\/13"],"award-info":[{"award-number":["1634\/13"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","award":["2017\/13"],"award-info":[{"award-number":["2017\/13"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003975","name":"Israel Cancer Association","doi-asserted-by":"publisher","award":["20150911"],"award-info":[{"award-number":["20150911"]}],"id":[{"id":"10.13039\/501100003975","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Israel Ministry of Health","award":["3-10146"],"award-info":[{"award-number":["3-10146"]}]},{"name":"Marie Curie International Reintegration","award":["618592"],"award-info":[{"award-number":["618592"]}]},{"name":"Runyon-Rachleff Innovator Award","award":["DRR-44-16"],"award-info":[{"award-number":["DRR-44-16"]}]},{"name":"Island Outreach Foundation"},{"DOI":"10.13039\/100001021","name":"Damon Runyon Cancer Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100001021","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Schaefer Research Scholars Program"},{"name":"Columbia University\u2019s College of Physicians and Surgeons","award":["CU16-1797"],"award-info":[{"award-number":["CU16-1797"]}]},{"DOI":"10.13039\/100006386","name":"Adenoid Cystic Carcinoma Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006386","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006386","name":"ACCRF","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006386","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,3,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>A major aim of single cell biology is to identify important cell types such as stem cells in heterogeneous tissues and tumors. This is typically done by isolating hundreds of individual cells and measuring expression levels of multiple genes simultaneously from each cell. Then, clustering algorithms are used to group together similar single-cell expression profiles into clusters, each representing a distinct cell type. However, many of these clusters result from overfitting, meaning that rather than representing biologically meaningful cell types, they describe the intrinsic \u2018noise\u2019 in gene expression levels due to limitations in experimental precision or the intrinsic randomness of biochemical cellular processes. Consequentially, these non-meaningful clusters are most sensitive to noise: a slight shift in gene expression levels due to a repeated measurement will rearrange the grouping of data points such that these clusters break up.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>To identify the biologically meaningful clusters we propose a \u2018cluster robustness score\u2019: We add increasing amounts of noise (zero mean and increasing variance) and check which clusters are most robust in the sense that they do not mix with their neighbors up to high levels of noise. We show that biologically meaningful cell clusters that were manually identified in previously published single cell expression datasets have high robustness scores. These scores are higher than what would be expected in corresponding randomized homogeneous datasets having the same expression level statistics. We believe that this scoring system provides a more automated way to identify cell types in heterogeneous tissues and tumors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/bty708","type":"journal-article","created":{"date-parts":[[2018,8,26]],"date-time":"2018-08-26T07:10:48Z","timestamp":1535267448000},"page":"962-971","source":"Crossref","is-referenced-by-count":18,"title":["A cluster robustness score for identifying cell subpopulations in single cell gene expression datasets from heterogeneous tissues and tumors"],"prefix":"10.1093","volume":"35","author":[{"given":"Itamar","family":"Kanter","sequence":"first","affiliation":[{"name":"Department of Bioengineering and Bar-Ilan Institute of Nanotechnology and Advanced Materials (BINA), Bar-Ilan University, Ramat Gan, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Piero","family":"Dalerba","sequence":"additional","affiliation":[{"name":"Department of Pathology and Cell Biology, Department of Medicine (Division of Digestive and Liver Diseases), Herbert Irving Comprehensive Cancer Center (HICCC), and the Columbia Stem Cell Initiative (CSCI), Columbia University, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4733-262X","authenticated-orcid":false,"given":"Tomer","family":"Kalisky","sequence":"additional","affiliation":[{"name":"Department of Bioengineering and Bar-Ilan Institute of Nanotechnology and Advanced Materials (BINA), Bar-Ilan University, Ramat Gan, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,8,28]]},"reference":[{"key":"2023013107261046800_bty708-B1","doi-asserted-by":"crossref","first-page":"3983","DOI":"10.1073\/pnas.0530291100","article-title":"Prospective identification of tumorigenic breast cancer cells","volume":"100","author":"Al-Hajj","year":"2003","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023013107261046800_bty708-B2","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1038\/nature06196","article-title":"Identification of stem cells in small intestine and colon by marker gene Lgr5","volume":"449","author":"Barker","year":"2007","journal-title":"Nature"},{"key":"2023013107261046800_bty708-B3","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1038\/ni.3368","article-title":"The heterogeneity of human CD127+ innate lymphoid cells revealed by single-cell RNA sequencing","volume":"17","author":"Bj\u00f6rklund","year":"2016","journal-title":"Nat. Immunol"},{"key":"2023013107261046800_bty708-B4","doi-asserted-by":"crossref","first-page":"730","DOI":"10.1038\/nm0797-730","article-title":"Human acute myeloid leukemia is organized as a hierarchy that originates from a primitive hematopoietic cell","volume":"3","author":"Bonnet","year":"1997","journal-title":"Nat. Med"},{"key":"2023013107261046800_bty708-B5","doi-asserted-by":"crossref","first-page":"3696","DOI":"10.1096\/fj.08-102590","article-title":"Identification of a tumor-initiating stem cell population in human renal carcinomas","volume":"22","author":"Bussolati","year":"2008","journal-title":"Faseb J"},{"key":"2023013107261046800_bty708-B6","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/j.devcel.2015.09.009","article-title":"Intrinsic age-dependent changes and cell-cell contacts regulate nephron progenitor lifespan","volume":"35","author":"Chen","year":"2015","journal-title":"Dev. Cell"},{"key":"2023013107261046800_bty708-B7","doi-asserted-by":"crossref","first-page":"790","DOI":"10.1109\/34.400568","article-title":"Mean shift, mode seeking, and clustering","volume":"17","author":"Cheng","year":"1995","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"2023013107261046800_bty708-B8","doi-asserted-by":"crossref","first-page":"10158","DOI":"10.1073\/pnas.0703478104","article-title":"Phenotypic characterization of human colorectal cancer stem cells","volume":"104","author":"Dalerba","year":"2007","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023013107261046800_bty708-B9","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1038\/nbt.2038","article-title":"Single-cell dissection of transcriptional heterogeneity in human colon tumors","volume":"29","author":"Dalerba","year":"2011","journal-title":"Nat. Biotechnol"},{"key":"2023013107261046800_bty708-B10","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1016\/j.cell.2018.05.061","article-title":"Recovering Gene Interactions from Single-Cell Data Using Data Diffusion","volume":"174","author":"van Dijk","year":"2018","journal-title":"Cell"},{"key":"2023013107261046800_bty708-B11","author":"Ester","year":"1996"},{"key":"2023013107261046800_bty708-B12","doi-asserted-by":"crossref","first-page":"1258367","DOI":"10.1126\/science.1258367","article-title":"Combinatorial labeling of single cells for gene expression cytometry","volume":"347","author":"Fan","year":"2015","journal-title":"Science"},{"key":"2023013107261046800_bty708-B13","doi-asserted-by":"crossref","first-page":"972","DOI":"10.1126\/science.1136800","article-title":"Clustering by passing messages between data points","volume":"315","author":"Frey","year":"2007","journal-title":"Science"},{"key":"2023013107261046800_bty708-B21","doi-asserted-by":"crossref","DOI":"10.1186\/s12859-018-2226-y","article-title":"DrImpute: Imputing dropout events in single cell RNA sequencing data","author":"Gong","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2023013107261046800_bty708-B14","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1016\/j.stem.2013.07.017","article-title":"Mapping cellular hierarchy by single-cell analysis of the cell surface repertoire","volume":"13","author":"Guo","year":"2013","journal-title":"Cell Stem Cell"},{"key":"2023013107261046800_bty708-B15","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1016\/j.devcel.2010.02.012","article-title":"Resolution of cell fate decisions revealed by single-cell gene expression analysis from zygote to blastocyst","volume":"18","author":"Guo","year":"2010","journal-title":"Dev. Cell"},{"key":"2023013107261046800_bty708-B16","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1016\/j.cell.2016.07.043","article-title":"The spectrum and regulatory landscape of intestinal innate lymphoid cells are shaped by the microbiome","volume":"166","author":"Gury-BenAri","year":"2016","journal-title":"Cell"},{"key":"2023013107261046800_bty708-B17","doi-asserted-by":"crossref","first-page":"77.","DOI":"10.1186\/s13059-016-0938-8","article-title":"CEL-Seq2: sensitive highly-multiplexed single-cell RNA-Seq","volume":"17","author":"Hashimshony","year":"2016","journal-title":"Genome Biol"},{"key":"2023013107261046800_bty708-B18","doi-asserted-by":"crossref","first-page":"776","DOI":"10.1126\/science.1247651","article-title":"Massively parallel single-cell RNA-seq for marker-free decomposition of tissues into cell types","volume":"343","author":"Jaitin","year":"2014","journal-title":"Science"},{"key":"2023013107261046800_bty708-B19","doi-asserted-by":"crossref","DOI":"10.1002\/9780470316801","volume-title":"Finding Groups in Data: An Introduction to Cluster Analysis (Wiley Series in Probability and Statistics)","author":"Kaufman","year":"1990","edition":"2005th edn"},{"key":"2023013107261046800_bty708-B20","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1016\/j.cell.2015.04.044","article-title":"Droplet barcoding for single-cell transcriptomics applied to embryonic stem cells","volume":"161","author":"Klein","year":"2015","journal-title":"Cell"},{"key":"2023013107261046800_bty708-B22","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1016\/j.cell.2016.09.027","article-title":"Molecular diversity of midbrain development in mouse, human and stem cells","volume":"167","author":"La Manno","year":"2016","journal-title":"Cell"},{"key":"2023013107261046800_bty708-B23","doi-asserted-by":"crossref","first-page":"1030","DOI":"10.1158\/0008-5472.CAN-06-2030","article-title":"Identification of pancreatic cancer stem cells","volume":"67","author":"Li","year":"2007","journal-title":"Cancer Res"},{"key":"2023013107261046800_bty708-B24","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.1038\/nprot.2013.109","article-title":"Single-molecule mRNA detection and counting in mammalian tissue","volume":"8","author":"Lyubimova","year":"2013","journal-title":"Nat. Protoc"},{"key":"2023013107261046800_bty708-B25","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1016\/j.cell.2015.05.002","article-title":"Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets","volume":"161","author":"Macosko","year":"2015","journal-title":"Cell"},{"key":"2023013107261046800_bty708-B26","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1073\/pnas.1013004108","article-title":"Mouse telomerase reverse transcriptase (mTert) expression marks slowly cycling intestinal stem cells","volume":"108","author":"Montgomery","year":"2011","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023013107261046800_bty708-B27","doi-asserted-by":"crossref","first-page":"1396","DOI":"10.1126\/science.1254257","article-title":"Single-cell RNA-seq highlights intratumoral heterogeneity in primary glioblastoma","volume":"344","author":"Patel","year":"2014","journal-title":"Science"},{"key":"2023013107261046800_bty708-B28","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1038\/nprot.2014.006","article-title":"Full-length RNA-seq from single cells using Smart-seq2","volume":"9","author":"Picelli","year":"2014","journal-title":"Nat. Protoc"},{"key":"2023013107261046800_bty708-B29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13059-015-0805-z","article-title":"ZIFA: dimensionality reduction for zero-inflated single-cell gene expression analysis","volume":"16","author":"Pierson","year":"2015","journal-title":"Genome Biol"},{"key":"2023013107261046800_bty708-B30","doi-asserted-by":"crossref","first-page":"1792","DOI":"10.1111\/j.1582-4934.2008.00607.x","article-title":"Developmental tumourigenesis: nCAM as a putative marker for the malignant renal stem\/progenitor cell population","volume":"13","author":"Pode-Shakked","year":"2009","journal-title":"J. Cell. Mol. Med"},{"key":"2023013107261046800_bty708-B31","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1002\/emmm.201201516","article-title":"The isolation and characterization of renal cancer initiating cells from human Wilms\u2019 tumour xenografts unveils new therapeutic targets","volume":"5","author":"Pode-Shakked","year":"2013","journal-title":"EMBO Mol. Med"},{"key":"2023013107261046800_bty708-B32","doi-asserted-by":"crossref","first-page":"973","DOI":"10.1073\/pnas.0610117104","article-title":"Identification of a subpopulation of cells with cancer stem cell properties in head and neck squamous cell carcinoma","volume":"104","author":"Prince","year":"2007","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023013107261046800_bty708-B33","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1053\/j.gastro.2012.02.006","article-title":"Identification of a cKit(+) colonic crypt base secretory cell that supports Lgr5(+) stem cells in mice","volume":"142","author":"Rothenberg","year":"2012","journal-title":"Gastroenterology"},{"key":"2023013107261046800_bty708-B34","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: a graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"J. Comput. Appl. Math"},{"key":"2023013107261046800_bty708-B35","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1038\/nprot.2012.021","article-title":"Microfluidic single-cell real-time PCR for comparative analysis of gene expression patterns","volume":"7","author":"Sanchez-Freire","year":"2012","journal-title":"Nat. Protoc"},{"key":"2023013107261046800_bty708-B36","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1038\/ng.165","article-title":"Bmi1 is expressed in vivo in intestinal stem cells","volume":"40","author":"Sangiorgi","year":"2008","journal-title":"Nat. Genet"},{"key":"2023013107261046800_bty708-B37","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1038\/nature04372","article-title":"Generation of a functional mammary gland from a single stem cell","volume":"439","author":"Shackleton","year":"2006","journal-title":"Nature"},{"key":"2023013107261046800_bty708-B38","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1038\/nmeth.4145","article-title":"Effective detection of variation in single-cell transcriptomes using MATQ-seq","volume":"14","author":"Sheng","year":"2017","journal-title":"Nat. Methods"},{"key":"2023013107261046800_bty708-B39","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1038\/nature03128","article-title":"Identification of human brain tumour initiating cells","volume":"432","author":"Singh","year":"2004","journal-title":"Nature"},{"key":"2023013107261046800_bty708-B40","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1126\/science.2898810","article-title":"Purification and characterization of mouse hematopoietic stem cells","volume":"241","author":"Spangrude","year":"1988","journal-title":"Science"},{"key":"2023013107261046800_bty708-B41","doi-asserted-by":"crossref","first-page":"993","DOI":"10.1038\/nature04496","article-title":"Purification and unique properties of mammary epithelial stem cells","volume":"439","author":"Stingl","year":"2006","journal-title":"Nature"},{"key":"2023013107261046800_bty708-B42","doi-asserted-by":"crossref","first-page":"2924.","DOI":"10.1038\/ncomms3924","article-title":"Early dynamic fate changes in haemogenic endothelium characterized at the single-cell level","volume":"4","author":"Swiers","year":"2013","journal-title":"Nat. Commun"},{"key":"2023013107261046800_bty708-B43","article-title":"Varying-censoring aware matrix factorization for single cell RNA-sequencing","volume":"166736","author":"Townes","year":"2017","journal-title":"bioRxiv"},{"key":"2023013107261046800_bty708-B44","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1038\/nature13173","article-title":"Reconstructing lineage hierarchies of the distal lung epithelium using single-cell RNA-seq","volume":"509","author":"Treutlein","year":"2014","journal-title":"Nature"},{"key":"2023013107261046800_bty708-B45","doi-asserted-by":"crossref","first-page":"eaah4573.","DOI":"10.1126\/science.aah4573","article-title":"Single-cell RNA-seq reveals new types of human blood dendritic cells, monocytes, and progenitors","volume":"356","author":"Villani","year":"2017","journal-title":"Science"},{"key":"2023013107261046800_bty708-B46","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1038\/nmeth.4207","article-title":"Visualization and analysis of single-cell RNA-seq data by kernel-based similarity learning","volume":"14","author":"Wang","year":"2017","journal-title":"Nat. Methods"},{"key":"2023013107261046800_bty708-B47","doi-asserted-by":"crossref","first-page":"e179.","DOI":"10.1093\/nar\/gkx828","article-title":"Linnorm: improved statistical analysis for single cell RNA-seq expression data","volume":"45","author":"Yip","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2023013107261046800_bty708-B48","doi-asserted-by":"crossref","first-page":"1138","DOI":"10.1126\/science.aaa1934","article-title":"Cell types in the mouse cortex and hippocampus revealed by single-cell RNA-seq","volume":"347","author":"Zeisel","year":"2015","journal-title":"Science"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/6\/962\/48966297\/bioinformatics_35_6_962.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/6\/962\/48966297\/bioinformatics_35_6_962.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T05:23:03Z","timestamp":1675142583000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/35\/6\/962\/5085371"}},"subtitle":[],"editor":[{"given":"Janet","family":"Kelso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2018,8,28]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2019,3,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bty708","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,3,15]]},"published":{"date-parts":[[2018,8,28]]}}}