{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T09:26:04Z","timestamp":1768814764788,"version":"3.49.0"},"reference-count":31,"publisher":"Oxford University Press (OUP)","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2006,5,15]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Motivation: Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data.<\/jats:p><jats:p>Results: Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.<\/jats:p><jats:p>Contact: \u00a0dp.berrar@ulster.ac.uk<\/jats:p>","DOI":"10.1093\/bioinformatics\/btl066","type":"journal-article","created":{"date-parts":[[2006,2,25]],"date-time":"2006-02-25T01:14:23Z","timestamp":1140830063000},"page":"1245-1250","source":"Crossref","is-referenced-by-count":44,"title":["Avoiding model selection bias in small-sample genomic datasets"],"prefix":"10.1093","volume":"22","author":[{"given":"Daniel","family":"Berrar","sequence":"first","affiliation":[{"name":"School of Biomedical Sciences, University of Ulster at Coleraine \u00a0 Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ian","family":"Bradbury","sequence":"additional","affiliation":[{"name":"School of Biomedical Sciences, University of Ulster at Coleraine \u00a0 Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Werner","family":"Dubitzky","sequence":"additional","affiliation":[{"name":"School of Biomedical Sciences, University of Ulster at Coleraine \u00a0 Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2006,2,24]]},"reference":[{"key":"2023012408203732900_b1","doi-asserted-by":"crossref","first-page":"6562","DOI":"10.1073\/pnas.102102699","article-title":"Selection bias in gene extraction on th basis of microarray gene expression data","volume":"98","author":"Ambroise","year":"2002","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012408203732900_b2","volume-title":"The Statistical Analysis of Data","author":"Anderson","year":"1986","edition":"2nd edn"},{"key":"2023012408203732900_b3","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1111\/j.2517-6161.1979.tb01066.x","article-title":"Reference posterior distributions for Bayesian inference","volume":"B41","author":"Bernado","year":"1979","journal-title":"J. R. Statist. Soc."},{"key":"2023012408203732900_b4","first-page":"5","article-title":"Multiclass cancer classification using gene expression profiling and probabilistic neural networks","author":"Berrar","year":"2003"},{"key":"2023012408203732900_b5","first-page":"3","article-title":"Evaluating the replicability of significance tests for comparing learning algorithms","author":"Bouckaert","year":"2004"},{"key":"2023012408203732900_b6","volume-title":"Classification and Regression Trees","author":"Breiman","year":"1984"},{"key":"2023012408203732900_b7","first-page":"321","article-title":"Multivariate functional interpolation and adaptive networks","volume":"2","author":"Broomhead","year":"1988","journal-title":"Complex Systems"},{"key":"2023012408203732900_b8","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1073\/pnas.97.1.262","article-title":"Knowledge-based analysis of microarray gene expression data by using support vector machines","volume":"97","author":"Brown","year":"2000","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012408203732900_b9","doi-asserted-by":"crossref","first-page":"1895","DOI":"10.1162\/089976698300017197","article-title":"Approximate statistical tests for comparing supervised classification learning algorithms","volume":"10","author":"Dietterich","year":"1998","journal-title":"Neural Comput."},{"key":"2023012408203732900_b10","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1198\/016214502753479248","article-title":"Comparison of discrimination methods for the classification of tumors using gene expression data","volume":"97","author":"Dudoit","year":"2002","journal-title":"J. Am. Stat. Assoc."},{"key":"2023012408203732900_b11","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm","year":"1979","journal-title":"Scand. J. Stat."},{"key":"2023012408203732900_b12","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1038\/89044","article-title":"Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks","volume":"7","author":"Khan","year":"2001","journal-title":"Nat. Med."},{"key":"2023012408203732900_b13","first-page":"223","article-title":"A study of cross-validation and bootstrap for accuracy estimation and model selection","author":"Kohavi","year":"1995"},{"key":"2023012408203732900_b14","doi-asserted-by":"crossref","first-page":"2429","DOI":"10.1093\/bioinformatics\/bth267","article-title":"A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression","volume":"20","author":"Li","year":"2004","journal-title":"Bioinformatics"},{"key":"2023012408203732900_b15","article-title":"High-resolution serum proteomic patterns for ovarian cancer detection. Letter to the editor","author":"Liotta"},{"key":"2023012408203732900_b16","doi-asserted-by":"crossref","first-page":"997","DOI":"10.1101\/gr.2156804","article-title":"Genomics, prior probability, and statistical tests of multiple hypotheses","volume":"14","author":"Manly","year":"2004","journal-title":"Genome Res."},{"key":"2023012408203732900_b17","article-title":"Small sample statistics for classification error rates II: confidence intervals and significance tests","author":"Martin","year":"1996"},{"key":"2023012408203732900_b18","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1023\/A:1024068626366","article-title":"Inference for generalization error","volume":"52","author":"Nadeau","year":"2003","journal-title":"Mach. Learn."},{"key":"2023012408203732900_b19","first-page":"43","article-title":"Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions","author":"Provost","year":"1998"},{"key":"2023012408203732900_b20","volume-title":"C4.5:Programs for Machine Learning","author":"Quinlan","year":"1993"},{"key":"2023012408203732900_b21","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1089\/106652702760138592","article-title":"A paradigm for class prediction using gene expression profiles","volume":"9","author":"Radmacher","year":"2002","journal-title":"J. Comput. Biol."},{"key":"2023012408203732900_b22","doi-asserted-by":"crossref","first-page":"15149","DOI":"10.1073\/pnas.211566398","article-title":"Multiclass cancer diagnosis using tumor gene expression signatures","volume":"98","author":"Ramaswamy","year":"2001","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012408203732900_b23","volume-title":"Fundamentals of Biostatistics","author":"Rosner","year":"2000","edition":"5th edn"},{"key":"2023012408203732900_b24","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1038\/73432","article-title":"Systematic variation in gene expression patterns in human cancer cell lines","volume":"24","author":"Ross","year":"2000","journal-title":"Nat. Gen."},{"key":"2023012408203732900_b25","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1038\/73439","article-title":"A gene expression database for the molecular pharmacology of cancer","volume":"24","author":"Scherf","year":"2000","journal-title":"Nat. Gen."},{"key":"2023012408203732900_b26","first-page":"263","article-title":"Class prediction and discovery using gene expression data","author":"Slonim","year":"2000"},{"key":"2023012408203732900_b27","doi-asserted-by":"crossref","first-page":"1484","DOI":"10.1093\/bioinformatics\/btg182","article-title":"Class prediction and discovery using gene microarray and proteomics mass spectroscopy data: curses, caveats, cautions","volume":"19","author":"Somorjai","year":"2003","journal-title":"Bioinformatics"},{"key":"2023012408203732900_b31","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1093\/bioinformatics\/bti033","article-title":"A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis","volume":"21","author":"Statnikov","year":"2005","journal-title":"Bioinformatics"},{"key":"2023012408203732900_b28","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/1471-2105-4-60","article-title":"Tumor classification and marker gene prediction by feature selection and fuzzy c-means clustering using microarray data","volume":"4","author":"Wang","year":"2003","journal-title":"BMC Bioinformatics"},{"key":"2023012408203732900_b29","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1016\/S1535-6108(02)00032-6","article-title":"Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling","volume":"1","author":"Yeoh","year":"2002","journal-title":"Cancer Cell"},{"key":"2023012408203732900_b30","doi-asserted-by":"crossref","first-page":"6730","DOI":"10.1073\/pnas.111153698","article-title":"Recursive partitioning for tumor classification with gene expression microarray data","volume":"98","author":"Zhang","year":"2001","journal-title":"Proc. Natl Acad. Sci. USA"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/22\/10\/1245\/48838245\/bioinformatics_22_10_1245.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/22\/10\/1245\/48838245\/bioinformatics_22_10_1245.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T21:38:51Z","timestamp":1736285931000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/22\/10\/1245\/236953"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2006,2,24]]},"references-count":31,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2006,5,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btl066","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2006,5,15]]},"published":{"date-parts":[[2006,2,24]]}}}