{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,9]],"date-time":"2025-11-09T07:33:41Z","timestamp":1762673621609,"version":"3.33.0"},"reference-count":22,"publisher":"Oxford University Press (OUP)","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2007,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Motivation: Gene expression data offer a large number of potentially useful predictors for the classification of tissue samples into classes, such as diseased and non-diseased. The predictive error rate of classifiers can be estimated using methods such as cross-validation. We have investigated issues of interpretation and potential bias in the reporting of error rate estimates. The issues considered here are optimization and selection biases, sampling effects, measures of misclassification rate, baseline error rates, two-level external cross-validation and a novel proposal for detection of bias using the permutation mean.<\/jats:p><jats:p>Results: Reporting an optimal estimated error rate incurs an optimization bias. Downward bias of 3\u20135% was found in an existing study of classification based on gene expression data and may be endemic in similar studies. Using a simulated non-informative dataset and two example datasets from existing studies, we show how bias can be detected through the use of label permutations and avoided using two-level external cross-validation. Some studies avoid optimization bias by using single-level cross-validation and a test set, but error rates can be more accurately estimated via two-level cross-validation. In addition to estimating the simple overall error rate, we recommend reporting class error rates plus where possible the conditional risk incorporating prior class probabilities and a misclassification cost matrix. We also describe baseline error rates derived from three trivial classifiers which ignore the predictors.<\/jats:p><jats:p>Availability: R code which implements two-level external cross-validation with the PAMR package, experiment code, dataset details and additional figures are freely available for non-commercial use from http:\/\/www.maths.qut.edu.au\/profiles\/wood\/permr.jsp<\/jats:p><jats:p>Contact: i.wood@qut.edu.au<\/jats:p><jats:p>Supplementary information: Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btm117","type":"journal-article","created":{"date-parts":[[2007,3,29]],"date-time":"2007-03-29T00:32:41Z","timestamp":1175128361000},"page":"1363-1370","source":"Crossref","is-referenced-by-count":63,"title":["Classification based upon gene expression data: bias and precision of error rates"],"prefix":"10.1093","volume":"23","author":[{"given":"Ian A.","family":"Wood","sequence":"first","affiliation":[{"name":"1 School of Mathematical Sciences, Queensland University of Technology, Gardens Point, GPO Box 2434, Brisbane, QLD 4001, Australia and 2Queensland Institute of Medical Research, Post Office, Royal Brisbane Hospital, 300 Herston Rd., Herston, QLD 4029, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter M.","family":"Visscher","sequence":"additional","affiliation":[{"name":"1 School of Mathematical Sciences, Queensland University of Technology, Gardens Point, GPO Box 2434, Brisbane, QLD 4001, Australia and 2Queensland Institute of Medical Research, Post Office, Royal Brisbane Hospital, 300 Herston Rd., Herston, QLD 4029, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kerrie L.","family":"Mengersen","sequence":"additional","affiliation":[{"name":"1 School of Mathematical Sciences, Queensland University of Technology, Gardens Point, GPO Box 2434, Brisbane, QLD 4001, Australia and 2Queensland Institute of Medical Research, Post Office, Royal Brisbane Hospital, 300 Herston Rd., Herston, QLD 4029, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2007,3,28]]},"reference":[{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"6562","DOI":"10.1073\/pnas.102102699","article-title":"Selection bias in gene extraction on the basis of microarray gene-expression data","volume":"99","author":"Ambroise","year":"2002","journal-title":"PNAS"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1093\/bioinformatics\/btg419","article-title":"Is cross-validation valid for small-sample microarray classification?","volume":"20","author":"Braga-Neto","year":"2004","journal-title":"Bioinformatics"},{"volume-title":"Classification and Regression Trees.","year":"1984","author":"Breiman","key":"2023041104584751300_"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"4148","DOI":"10.1093\/bioinformatics\/bti681","article-title":"Classification of microarrays to nearest centroids","volume":"21","author":"Dabney","year":"2005","journal-title":"Bioinformatics"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1201\/9780203011232.ch3","article-title":"Classification in microarray experiments","volume-title":"Statistical Analysis of Gene Expression Microarray Data","author":"Dudoit","year":"2003"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1080\/01621459.1983.10477973","article-title":"Estimating the error rate of a prediction rule: improvement on cross-validation","volume":"78","author":"Efron","year":"1983","journal-title":"J. Am. Stat. Assoc"},{"key":"2023041104584751300_","first-page":"548","article-title":"Improvements on cross-validation: The .632+ bootstrap method","volume":"92","author":"Efron","year":"1997","journal-title":"J. Am. Stat. Assoc"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4757-2346-5","volume-title":"Permutation Tests: a Practical Guide to Resampling Methods for Testing Hypotheses","author":"Good","year":"1994"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1023\/A:1012487302797","article-title":"Gene selection for cancer classification using support vector machines","volume":"46","author":"Guyon","year":"2002","journal-title":"Machine Learning"},{"key":"2023041104584751300_","volume-title":"Continuous Univariate Distributions","author":"Johnson","year":"1994","edition":"2nd edn"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1038\/89044","article-title":"Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks","volume":"7","author":"Khan","year":"2001","journal-title":"Nature Med"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","DOI":"10.1002\/0471725293","volume-title":"Discriminant Analysis and Statistical Pattern Recognition","author":"McLachlan","year":"1992"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","DOI":"10.1002\/047172842X","volume-title":"Analyzing Microarray Gene Expression Data","author":"McLachlan","year":"2004"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"3301","DOI":"10.1093\/bioinformatics\/bti499","article-title":"Prediction error estimation: a comparison of resampling methods","volume":"21","author":"Molinaro","year":"2005","journal-title":"Bioinformatics"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"R634","DOI":"10.1186\/bcr1203","article-title":"Early detection of breast cancer based on gene-expression patterns in peripheral blood cells","volume":"7","author":"Sharma","year":"2005","journal-title":"Breast Cancer Res"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","article-title":"Cross-validatory choice and assessment of statistical predictions","volume":"36","author":"Stone","year":"1974","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1093\/bioinformatics\/bti033","article-title":"A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis","volume":"21","author":"Statnikov","year":"2005","journal-title":"Bioinformatics"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"6567","DOI":"10.1073\/pnas.082099299","article-title":"Diagnosis of multiple cancer types by shrunken centroids of gene expression","volume":"99","author":"Tibshirani","year":"2002","journal-title":"PNAS"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1214\/ss\/1056397488","article-title":"Class prediction by nearest shrunken centroids, with applications to DNA microarrays","volume":"18","author":"Tibshirani","year":"2003","journal-title":"Stat. Sci"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/1471-2105-7-91","article-title":"Bias in error estimation when using cross-validation for model selection","volume":"7","author":"Varma","year":"2006","journal-title":"BMC Bioinformatics"},{"key":"2023041104584751300_","doi-asserted-by":"crossref","first-page":"3755","DOI":"10.1093\/bioinformatics\/bti429","article-title":"A protocol for building and evaluating predictors of disease state based on microarray data","volume":"21","author":"Wessels","year":"2005","journal-title":"Bioinformatics"},{"key":"2023041104584751300_","article-title":"On selection biases with prediction rules formed from gene expression data","author":"Zhu","year":"2007","journal-title":"J. Stat. Plan. Inference"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/23\/11\/1363\/49813873\/bioinformatics_23_11_1363.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/23\/11\/1363\/49813873\/bioinformatics_23_11_1363.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,15]],"date-time":"2025-01-15T07:53:20Z","timestamp":1736927600000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/23\/11\/1363\/201367"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2007,3,28]]},"references-count":22,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2007,6,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btm117","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"type":"electronic","value":"1367-4811"},{"type":"print","value":"1367-4803"}],"subject":[],"published-other":{"date-parts":[[2007,6]]},"published":{"date-parts":[[2007,3,28]]}}}