{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T15:38:51Z","timestamp":1792597131563,"version":"4.5.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"S15","license":[{"start":{"date-parts":[[2019,12,1]],"date-time":"2019-12-01T00:00:00Z","timestamp":1575158400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,24]],"date-time":"2019-12-24T00:00:00Z","timestamp":1577145600000},"content-version":"vor","delay-in-days":23,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Cluster analysis is a core task in modern data-centric computation. Algorithmic choice is driven by factors such as data size and heterogeneity, the similarity measures employed, and the type of clusters sought. Familiarity and mere preference often play a significant role as well. Comparisons between clustering algorithms tend to focus on cluster quality. Such comparisons are complicated by the fact that algorithms often have multiple settings that can affect the clusters produced. Such a setting may represent, for example, a preset variable, a parameter of interest, or various sorts of initial assignments. A question of interest then is this: to what degree do the clusters produced vary as setting values change?<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>This work introduces a new metric, termed simply \u201crobustness\u201d, designed to answer that question. Robustness is an easily-interpretable measure of the propensity of a clustering algorithm to maintain output coherence over a range of settings. The robustness of eleven popular clustering algorithms is evaluated over some two dozen publicly available mRNA expression microarray datasets. Given their straightforwardness and predictability, hierarchical methods generally exhibited the highest robustness on most datasets. Of the more complex strategies, the paraclique algorithm yielded consistently higher robustness than other algorithms tested, approaching and even surpassing hierarchical methods on several datasets. Other techniques exhibited mixed robustness, with no clear distinction between them.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>Robustness provides a simple and intuitive measure of the stability and predictability of a clustering algorithm. It can be a useful tool to aid both in algorithm selection and in deciding how much effort to devote to parameter tuning.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s12859-019-3089-6","type":"journal-article","created":{"date-parts":[[2019,12,24]],"date-time":"2019-12-24T09:02:35Z","timestamp":1577178155000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["A robustness metric for biological data clustering algorithms"],"prefix":"10.1186","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8113-7652","authenticated-orcid":false,"given":"Yuping","family":"Lu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charles A.","family":"Phillips","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael A.","family":"Langston","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,12,24]]},"reference":[{"key":"3089_CR1","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/1471-2105-13-S10-S7","volume":"13","author":"JJ Jay","year":"2012","unstructured":"Jay JJ, Eblen JD, Zhang Y, Benson M, Perkins AD, Saxton AM, Voy BH, Chesler EJ, Langston MA. A systematic comparison of genome-scale clustering algorithms. BMC Bioinformatics. 2012; 13:7.","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"3089_CR2","first-page":"241","volume":"12","author":"G Chen","year":"2002","unstructured":"Chen G, Jaradat SA, Banerjee N, Tanaka TS, Ko MS, Zhang MQ. Evaluation and comparison of clustering algorithms in analyzing es cell gene expression data. Stat Sin. 2002; 12(1):241\u201362.","journal-title":"Stat Sin"},{"key":"3089_CR3","doi-asserted-by":"crossref","unstructured":"Datta S, Datta S. BMC Bioinformatics. 2006; 7(1):397.","DOI":"10.1186\/1471-2105-7-397"},{"issue":"43","key":"3089_CR4","doi-asserted-by":"publisher","first-page":"15545","DOI":"10.1073\/pnas.0506580102","volume":"102","author":"A Subramanian","year":"2005","unstructured":"Subramanian A, Tamayo P, Mootha VK, Mukherjee S, Ebert BL, Gillette MA, Paulovich A, Pomeroy SL, Golub TR, Lander ES, et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci. 2005; 102(43):15545\u201350.","journal-title":"Proc Natl Acad Sci"},{"key":"3089_CR5","doi-asserted-by":"crossref","unstructured":"Huang DW, Sherman BT, Tan Q, Kir J, Liu D, Bryant D, Guo Y, Stephens R, Baseler MW, Lane HC, et al. Nucleic Acids Res. 2007; 35(suppl_2):169\u201375.","DOI":"10.1093\/nar\/gkm415"},{"key":"3089_CR6","doi-asserted-by":"crossref","unstructured":"De Vries GK, Van Hage WR, Van Someren M. Comparing vessel trajectories using geographical domain knowledge and alignments. In: Data Mining Workshops (ICDMW), 2010 IEEE International Conference On. IEEE: 2010. p. 209\u201316.","DOI":"10.1109\/ICDMW.2010.123"},{"issue":"7","key":"3089_CR7","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1016\/S0262-8856(01)00089-0","volume":"20","author":"M Liu","year":"2002","unstructured":"Liu M, Samal A. Cluster validation using legacy delineations. Image Vis Comput. 2002; 20(7):459\u201367.","journal-title":"Image Vis Comput"},{"issue":"23","key":"3089_CR8","doi-asserted-by":"publisher","first-page":"8577","DOI":"10.1073\/pnas.0601602103","volume":"103","author":"ME Newman","year":"2006","unstructured":"Newman ME. Modularity and community structure in networks. Proc Natl Acad Sci. 2006; 103(23):8577\u201382.","journal-title":"Proc Natl Acad Sci"},{"issue":"2","key":"3089_CR9","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1007\/BF02289146","volume":"14","author":"RD Luce","year":"1949","unstructured":"Luce RD, Perry AD. A method of matrix analysis of group structure. Psychometrika. 1949; 14(2):95\u2013116.","journal-title":"Psychometrika"},{"key":"3089_CR10","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511815478","volume-title":"Social Network Analysis: Methods and Applications. vol. 8","author":"S Wasserman","year":"1994","unstructured":"Wasserman S, Faust K. Social Network Analysis: Methods and Applications. vol. 8. Cambridge: Cambridge university press; 1994."},{"key":"3089_CR11","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","volume":"20","author":"PJ Rousseeuw","year":"1987","unstructured":"Rousseeuw PJ. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math. 1987; 20:53\u201365.","journal-title":"J Comput Appl Math"},{"issue":"336","key":"3089_CR12","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1080\/01621459.1971.10482356","volume":"66","author":"WM Rand","year":"1971","unstructured":"Rand WM. Objective criteria for the evaluation of clustering methods. J Am Stat Assoc. 1971; 66(336):846\u201350.","journal-title":"J Am Stat Assoc"},{"issue":"1-3","key":"3089_CR13","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1007\/BF02614317","volume":"79","author":"P Hansen","year":"1997","unstructured":"Hansen P, Jaumard B. Cluster analysis and mathematical programming. Math Program. 1997; 79(1-3):191\u2013215.","journal-title":"Math Program"},{"issue":"1","key":"3089_CR14","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/BF02291174","volume":"38","author":"L Hubert","year":"1973","unstructured":"Hubert L. Min and max hierarchical clustering using asymmetric similarity measures. Psychometrika. 1973; 38(1):63\u201372.","journal-title":"Psychometrika"},{"key":"3089_CR15","unstructured":"Rosenberg A, Hirschberg J. V-measure: A conditional entropy-based external cluster evaluation measure. EMNLP-CoNLL 2007. 2007:410."},{"key":"3089_CR16","unstructured":"Vinh NX, Epps J, Bailey J. J Mach Learn Res. 2010; 11(Oct):2837\u201354."},{"key":"3089_CR17","doi-asserted-by":"crossref","unstructured":"Edgar R, Domrachev M, Lash AE. Nucleic Acids Res. 2002; 30(1):207\u201310.","DOI":"10.1093\/nar\/30.1.207"},{"key":"3089_CR18","doi-asserted-by":"crossref","unstructured":"Huttenhower C, Flamholz AI, Landis JN, Sahi S, Myers CL, Olszewski KL, Hibbs MA, Siemers NO, Troyanskaya OG, Coller HA. BMC Bioinformatics. 2007; 8(1):250.","DOI":"10.1186\/1471-2105-8-250"},{"issue":"3","key":"3089_CR19","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1007\/BF02289588","volume":"32","author":"SC Johnson","year":"1967","unstructured":"Johnson SC. Hierarchical clustering schemes. Psychometrika. 1967; 32(3):241\u201354.","journal-title":"Psychometrika"},{"issue":"1","key":"3089_CR20","first-page":"100","volume":"28","author":"JA Hartigan","year":"1979","unstructured":"Hartigan JA, Wong MA. Algorithm as 136: A k-means clustering algorithm. J R Stat Soc Ser C (Appl Stat). 1979; 28(1):100\u20138.","journal-title":"J R Stat Soc Ser C (Appl Stat)"},{"issue":"11","key":"3089_CR21","doi-asserted-by":"publisher","first-page":"1106","DOI":"10.1101\/gr.9.11.1106","volume":"9","author":"LJ Heyer","year":"1999","unstructured":"Heyer LJ, Kruglyak S, Yooseph S. Exploring expression data: identification and analysis of coexpressed genes. Genome Res. 1999; 9(11):1106\u201315.","journal-title":"Genome Res"},{"key":"3089_CR22","volume-title":"Systems Biology and Regulatory Genomics","author":"EJ Chesler","year":"2007","unstructured":"Chesler EJ, Langston MA. Combinatorial genetic regulatory network analysis tools for high throughput transcriptomic data. In: Systems Biology and Regulatory Genomics. Berlin, Heidelberg: Springer: 2007. p. 150\u201365."},{"key":"3089_CR23","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1016\/j.dam.2015.11.010","volume":"204","author":"RD Hagan","year":"2016","unstructured":"Hagan RD, Langston MA, Wang K. Lower bounds on paraclique density. Discret Appl Math. 2016; 204:208\u201312.","journal-title":"Discret Appl Math"},{"issue":"14","key":"3089_CR24","doi-asserted-by":"publisher","first-page":"1787","DOI":"10.1093\/bioinformatics\/btg232","volume":"19","author":"R Sharan","year":"2003","unstructured":"Sharan R, Maron-Katz A, Shamir R. Click and expander: a system for clustering and visualizing gene expression data. Bioinformatics. 2003; 19(14):1787\u201399.","journal-title":"Bioinformatics"},{"key":"3089_CR25","doi-asserted-by":"crossref","unstructured":"Zhang B, Horvath S. A general framework for weighted gene co-expression network analysis. Stat Appl Genet Mol Biol. 2005;4(1).","DOI":"10.2202\/1544-6115.1128"},{"issue":"6","key":"3089_CR26","doi-asserted-by":"publisher","first-page":"2907","DOI":"10.1073\/pnas.96.6.2907","volume":"96","author":"P Tamayo","year":"1999","unstructured":"Tamayo P, Slonim D, Mesirov J, Zhu Q, Kitareewan S, Dmitrovsky E, Lander ES, Golub TR. Interpreting patterns of gene expression with self-organizing maps: methods and application to hematopoietic differentiation. Proc Natl Acad Sci. 1999; 96(6):2907\u201312.","journal-title":"Proc Natl Acad Sci"},{"issue":"3","key":"3089_CR27","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1016\/0360-8352(89)90160-5","volume":"16","author":"HK Seifoddini","year":"1989","unstructured":"Seifoddini HK. Single linkage versus average linkage clustering in machine cells formation applications. Comput Ind Eng. 1989; 16(3):419\u201326.","journal-title":"Comput Ind Eng"},{"issue":"5","key":"3089_CR28","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1007\/s00500-003-0346-3","volume":"9","author":"P Dawyndt","year":"2005","unstructured":"Dawyndt P, De Meyer H, De Baets B. The complete linkage clustering algorithm revisited. Soft Comput. 2005; 9(5):385\u201392.","journal-title":"Soft Comput"},{"key":"3089_CR29","doi-asserted-by":"crossref","unstructured":"McQuitty LL. Educ Psychol Meas. 1966; 26(4):825\u201331.","DOI":"10.1177\/001316446602600402"},{"issue":"301","key":"3089_CR30","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1080\/01621459.1963.10500845","volume":"58","author":"JH Ward Jr","year":"1963","unstructured":"Ward Jr JH. Hierarchical grouping to optimize an objective function. J Am Stat Assoc. 1963; 58(301):236\u201344.","journal-title":"J Am Stat Assoc"},{"issue":"7","key":"3089_CR31","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1371\/journal.pcbi.0020089","volume":"2","author":"BH Voy","year":"2006","unstructured":"Voy BH, Scharff JA, Perkins AD, Saxton AM, Borate B, Chesler EJ, Branstetter LK, Langston MA. Extracting gene networks for low-dose radiation using graph theoretical algorithms. PLoS Comput Biol. 2006; 2(7):89.","journal-title":"PLoS Comput Biol"},{"key":"3089_CR32","doi-asserted-by":"publisher","first-page":"20837","DOI":"10.1038\/srep20837","volume":"6","author":"E Willems","year":"2016","unstructured":"Willems E, Guerrero-Bosagna C, Decuypere E, Janssens S, Buyse J, Buys N, Jensen P, Everaert N. Differential expression of genes and dna methylation associated with prenatal protein undernutrition by albumen removal in an avian model. Sci Rep. 2016; 6:20837.","journal-title":"Sci Rep"},{"issue":"1","key":"3089_CR33","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1186\/s12920-015-0088-y","volume":"8","author":"I Herrer","year":"2015","unstructured":"Herrer I, Rosell\u00f3-Llet\u00ed E, Ortega A, Taraz\u00f3n E, Molina-Navarro MM, Trivi\u00f1o JC, Mart\u00ednez-Dolz L, Almenar L, Lago F, S\u00e1nchez-L\u00e1zaro I, et al. Gene expression network analysis reveals new transcriptional regulators as novel factors in human ischemic cardiomyopathy. BMC Med Genomics. 2015; 8(1):14.","journal-title":"BMC Med Genomics"},{"issue":"1","key":"3089_CR34","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1007\/s12284-010-9037-8","volume":"3","author":"R Venu","year":"2010","unstructured":"Venu R, Madhav MS, Sreerekha M, Nobuta K, Zhang Y, Carswell P, Boehm MJ, Meyers BC, Korth KL, Wang G-L. Deep and comparative transcriptome analysis of rice plants infested by the beet armyworm (spodoptera exigua) and water weevil (lissorhoptrus oryzophilus). Rice. 2010; 3(1):22\u201335.","journal-title":"Rice"},{"key":"3089_CR35","doi-asserted-by":"crossref","unstructured":"Perkins AD, Langston MA. Threshold selection in gene co-expression networks using spectral graph theory techniques. In: BMC Bioinformatics. BioMed Central: 2009. p. 4.","DOI":"10.1186\/1471-2105-10-S11-S4"},{"key":"3089_CR36","unstructured":"MacQueen J, et al. Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1. Oakland: 1967. p. 281\u201397."},{"key":"3089_CR37","unstructured":"R Core Team. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing; 2017. https:\/\/www.R-project.org. Accessed 11 Jul 2017."},{"issue":"10","key":"3089_CR38","doi-asserted-by":"publisher","first-page":"2987","DOI":"10.1091\/mbc.12.10.2987","volume":"12","author":"AP Gasch","year":"2001","unstructured":"Gasch AP, Huang M, Metzner S, Botstein D, Elledge SJ, Brown PO. Genomic expression responses to dna-damaging agents and the regulatory role of the yeast atr homolog mec1p. Mol Biol Cell. 2001; 12(10):2987\u20133003.","journal-title":"Mol Biol Cell"},{"key":"3089_CR39","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1186\/1471-2105-13-S10-S5","volume":"13","author":"JD Eblen","year":"2012","unstructured":"Eblen JD, Phillips CA, Rogers GL, Langston MA. The maximum clique enumeration problem: algorithms, applications, and implementations. BMC Bioinformatics. 2012; 13:5.","journal-title":"BMC Bioinformatics"},{"issue":"2","key":"3089_CR40","first-page":"48","volume":"3","author":"A Baratloo","year":"2015","unstructured":"Baratloo A, Hosseini M, Negida A, El Ashal G. Part 1: simple definition and calculation of accuracy, sensitivity and specificity. Emergency. 2015; 3(2):48\u201349.","journal-title":"Emergency"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-019-3089-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12859-019-3089-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-019-3089-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,23]],"date-time":"2020-12-23T00:05:59Z","timestamp":1608681959000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-019-3089-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12]]},"references-count":40,"journal-issue":{"issue":"S15","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["3089"],"URL":"https:\/\/doi.org\/10.1186\/s12859-019-3089-6","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12]]},"assertion":[{"value":"23 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 September 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 December 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"503"}}