{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T13:59:14Z","timestamp":1783087154596,"version":"3.54.6"},"reference-count":30,"publisher":"Oxford University Press (OUP)","issue":"15","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2005,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: In genomic studies, thousands of features are collected on relatively few samples. One of the goals of these studies is to build classifiers to predict the outcome of future observations. There are three inherent steps to this process: feature selection, model selection and prediction assessment. With a focus on prediction assessment, we compare several methods for estimating the \u2018true\u2019 prediction error of a prediction model in the presence of feature selection.<\/jats:p>\n               <jats:p>Results: For small studies where features are selected from thousands of candidates, the resubstitution and simple split-sample estimates are seriously biased. In these small samples, leave-one-out cross-validation (LOOCV), 10-fold cross-validation (CV) and the .632+ bootstrap have the smallest bias for diagonal discriminant analysis, nearest neighbor and classification trees. LOOCV and 10-fold CV have the smallest bias for linear discriminant analysis. Additionally, LOOCV, 5- and 10-fold CV, and the .632+ bootstrap have the lowest mean square error. The .632+ bootstrap is quite biased in small sample sizes with strong signal-to-noise ratios. Differences in performance among resampling methods are reduced as the number of specimens available increase.<\/jats:p>\n               <jats:p>Contact: \u00a0annette.molinaro@yale.edu<\/jats:p>\n               <jats:p>Supplementary Information: A complete compilation of results and R code for simulations and analyses are available in Molinaro et al. (2005) (http:\/\/linus.nci.nih.gov\/brb\/TechReport.htm).<\/jats:p>","DOI":"10.1093\/bioinformatics\/bti499","type":"journal-article","created":{"date-parts":[[2005,5,20]],"date-time":"2005-05-20T00:24:12Z","timestamp":1116548652000},"page":"3301-3307","source":"Crossref","is-referenced-by-count":1113,"title":["Prediction error estimation: a comparison of resampling methods"],"prefix":"10.1093","volume":"21","author":[{"given":"Annette M.","family":"Molinaro","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Simon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruth M.","family":"Pfeiffer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2005,5,19]]},"reference":[{"key":"2023051612004326900_B1","doi-asserted-by":"crossref","unstructured":"Bhattacharjee, A., et al. 2001Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proc. Natl Acad. Sci. USA  \u00a098\n\u00a013790\u201313795","DOI":"10.1073\/pnas.191502998"},{"key":"2023051612004326900_B2","doi-asserted-by":"crossref","unstructured":"Braga-Neto, U.M. and Dougherty, E.R. 2004Is cross-validation for small-sample microarray classification? Bioinformatics  \u00a020\n\u00a0374\u2013380","DOI":"10.1093\/bioinformatics\/btg419"},{"key":"2023051612004326900_B3","doi-asserted-by":"crossref","unstructured":"Breiman, L. and Spector, P. 1992Submodel selection and evaluation in regression. The X-random case. Int. Stat. Rev.  \u00a060\n\u00a0291\u2013319","DOI":"10.2307\/1403680"},{"key":"2023051612004326900_B4","unstructured":"Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J. Classification and Regression Trees \u00a01984, Monterey, CA  Wadsworth and Brooks\/Cole"},{"key":"2023051612004326900_B5","doi-asserted-by":"crossref","unstructured":"Bura, E. and Pfeiffer, R.M. 2003Graphical methods for class prediction using dimension reduction techniques on DNA microarray data. Bioinformatics  \u00a019\n\u00a01252\u20131258","DOI":"10.1093\/bioinformatics\/btg150"},{"key":"2023051612004326900_B6","doi-asserted-by":"crossref","unstructured":"Burman, P. 1989A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods. Biometrika  \u00a076\n\u00a0503\u2013514","DOI":"10.1093\/biomet\/76.3.503"},{"key":"2023051612004326900_B7","unstructured":"Davison, A.C. and Hinkley, D.V. Bootstrap Methods and Their Application \u00a01997, Cambridge, UK  Cambridge Series in Statistical and Probabilistic Mathematics Cambridge University Press"},{"key":"2023051612004326900_B8","unstructured":"Dettling, M. and Maechler, M. 2005Software R Contributed Package: Supclust: Supervised Clustering of Genes.   (http:\/\/cran.r-project.org) version 1.0-5"},{"key":"2023051612004326900_B9","unstructured":"Technical Report 126 Dudoit, S. and van der Laan, M.J. 2003Asymptotics of cross-validated risk estimation in model selection and performance assessment.   U.C. Berkeley Division of Biostatistics Working Paper Series. http:\/\/www.bepress.com\/ucbbiostat\/paper126"},{"key":"2023051612004326900_B10","doi-asserted-by":"crossref","unstructured":"Efron, B. 1983Estimating the error rate of a prediction rule: improvement on cross-validation. J. Am. Stat. Assoc.  \u00a078\n\u00a0316\u2013331","DOI":"10.1080\/01621459.1983.10477973"},{"key":"2023051612004326900_B11","doi-asserted-by":"crossref","unstructured":"Efron, B. 2004The estimation of prediction error: covariance penalties and cross-validation. J. Am. Stat. Assoc.  \u00a099\n\u00a0619\u2013642","DOI":"10.1198\/016214504000000692"},{"key":"2023051612004326900_B12","doi-asserted-by":"crossref","unstructured":"Efron, B. and Tibshirani, R.J. An Introduction to the Bootstrap \u00a01993, NY  Monographs on Statistics and Applied Probability Chapman and Hall Vol. 57","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"2023051612004326900_B13","doi-asserted-by":"crossref","unstructured":"Efron, B. and Tibshirani, R.J. 1997Improvements on cross-validation: the .632+ bootstrap method. J. Am. Stat. Assoc.  \u00a092\n\u00a0548\u2013560","DOI":"10.1080\/01621459.1997.10474007"},{"key":"2023051612004326900_B14","doi-asserted-by":"crossref","unstructured":"Geisser, S. 1975The predictive sample reuse method with applications. J. Am. Stat. Assoc.  \u00a070\n\u00a0320\u2013328","DOI":"10.1080\/01621459.1975.10479865"},{"key":"2023051612004326900_B15","unstructured":"Hastie, T., Tibshirani, R., Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction \u00a02003, New York  Springer Series in Statistics. Springer, 1st edn, 3rd print Springer-Verlag"},{"key":"2023051612004326900_B16","doi-asserted-by":"crossref","unstructured":"Ihaka, R. and Gentlemen, R. 1996R: a language for data analysis and graphics. J. Comput. Graph. Stat.  \u00a05\n\u00a0299\u2013314","DOI":"10.1080\/10618600.1996.10474713"},{"key":"2023051612004326900_B17","doi-asserted-by":"crossref","unstructured":"Lachenbruch, P.A. and Mickey, M.R. 1968Estimation of error rates in discriminant analysis. Technometrics  \u00a010\n\u00a01\u201311","DOI":"10.1080\/00401706.1968.10490530"},{"key":"2023051612004326900_B18","doi-asserted-by":"crossref","unstructured":"McLachlan, G.J. Discriminant Analysis and Statistical Pattern Recognition \u00a01992, New York  John Wiley & Sons Inc","DOI":"10.1002\/0471725293"},{"key":"2023051612004326900_B19","doi-asserted-by":"crossref","unstructured":"Molinaro, A.M., et al. 2004Tree-based multivariate regression and density estimation with right-censored data. J. Multivar. Anal.  \u00a090\n\u00a0154\u2013177","DOI":"10.1016\/j.jmva.2004.02.003"},{"key":"2023051612004326900_B20","unstructured":"Technical Report Number 30 Molinaro, A.M., Simon, R., Pfeiffer, R.M. 2005Prediction error estimation: a comparison of resampling methods.   Biometrics Research Branch, Division of Cancer Treatment and Diagnosis, NCI"},{"key":"2023051612004326900_B21","unstructured":"Quackenbush, J. 2004Meeting the challenges of functional genomics: from the laboratory to the clinic. Preclinica  \u00a02\n\u00a0313\u2013316"},{"key":"2023051612004326900_B22","doi-asserted-by":"crossref","unstructured":"Ransohoff, D.F. 2004Rules of evidence for cancer molecular marker discovery and validation. Nature Reviews\/Cancer  \u00a04\n\u00a0309\u2013313","DOI":"10.1038\/nrc1322"},{"key":"2023051612004326900_B23","doi-asserted-by":"crossref","unstructured":"Ripley, B.D. Pattern Recognition and Neural Networks \u00a01996, New York  Cambridge University Press","DOI":"10.1017\/CBO9780511812651"},{"key":"2023051612004326900_B24","unstructured":"Rosenwald, A., et al. 2002The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma. New Engl. J. Med.  \u00a0346\n\u00a01937\u20131946"},{"key":"2023051612004326900_B25","doi-asserted-by":"crossref","unstructured":"Simon, R., et al. 2003Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. J. Natl Cancer Inst.  \u00a095\n\u00a014\u201318","DOI":"10.1093\/jnci\/95.1.14"},{"key":"2023051612004326900_B26","doi-asserted-by":"crossref","unstructured":"Stone, M. 1974Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc., Series B  \u00a036\n\u00a0111\u2013147","DOI":"10.1111\/j.2517-6161.1974.tb00994.x"},{"key":"2023051612004326900_B27","doi-asserted-by":"crossref","unstructured":"Stone, M. 1977Asymptotics for and against cross-validation. Biometrika  \u00a064\n\u00a029\u201335","DOI":"10.1093\/biomet\/64.1.29"},{"key":"2023051612004326900_B28","unstructured":"Technical Report 61 Therneau, T. and Atkinson, E. 1997An introduction to recursive partitioning using the RPART routine.   Section of Biostatistics, Mayo Clinic, Rochester"},{"key":"2023051612004326900_B29","doi-asserted-by":"crossref","unstructured":"Venables, W.N. and Ripley, B.D. Modern Applied Statistics with S-PLUS \u00a01994, New York  Springer","DOI":"10.1007\/978-1-4899-2819-1"},{"key":"2023051612004326900_B30","doi-asserted-by":"crossref","unstructured":"Wright, G., et al. 2003A gene expression-based method to diagnose clinically distinct subgroups of diffuse large B cell lymphoma. Proc. Natl Acad. Sci. USA  \u00a0100\n\u00a09991\u20139996","DOI":"10.1073\/pnas.1732008100"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/21\/15\/3301\/50340684\/bioinformatics_21_15_3301.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/21\/15\/3301\/50340684\/bioinformatics_21_15_3301.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,16]],"date-time":"2023-05-16T12:01:56Z","timestamp":1684238516000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/21\/15\/3301\/195433"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2005,5,19]]},"references-count":30,"journal-issue":{"issue":"15","published-print":{"date-parts":[[2005,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bti499","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2005,8]]},"published":{"date-parts":[[2005,5,19]]}}}