{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T01:57:52Z","timestamp":1762999072660},"reference-count":15,"publisher":"Oxford University Press (OUP)","issue":"22","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2014,11,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Supervised machine learning is commonly applied in genomic research to construct a classifier from the training data that is generalizable to predict independent testing data. When test datasets are not available, cross-validation is commonly used to estimate the error rate. Many machine learning methods are available, and it is well known that no universally best method exists in general. It has been a common practice to apply many machine learning methods and report the method that produces the smallest cross-validation error rate. Theoretically, such a procedure produces a selection bias. Consequently, many clinical studies with moderate sample sizes (e.g. n = 30\u201360) risk reporting a falsely small cross-validation error rate that could not be validated later in independent cohorts.<\/jats:p>\n               <jats:p>Results: In this article, we illustrated the probabilistic framework of the problem and explored the statistical and asymptotic properties. We proposed a new bias correction method based on learning curve fitting by inverse power law (IPL) and compared it with three existing methods: nested cross-validation, weighted mean correction and Tibshirani-Tibshirani procedure. All methods were compared in simulation datasets, five moderate size real datasets and two large breast cancer datasets. The result showed that IPL outperforms the other methods in bias correction with smaller variance, and it has an additional advantage to extrapolate error estimates for larger sample sizes, a practical feature to recommend whether more samples should be recruited to improve the classifier and accuracy. An R package \u2018MLbias\u2019 and all source files are publicly available.<\/jats:p>\n               <jats:p>Availability and implementation: tsenglab.biostat.pitt.edu\/software.htm.<\/jats:p>\n               <jats:p>Contact: \u00a0ctseng@pitt.edu<\/jats:p>\n               <jats:p>Supplementary information: \u00a0Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btu520","type":"journal-article","created":{"date-parts":[[2014,8,2]],"date-time":"2014-08-02T07:42:41Z","timestamp":1406965361000},"page":"3152-3158","source":"Crossref","is-referenced-by-count":24,"title":["Bias correction for selecting the minimal-error classifier from many machine learning models"],"prefix":"10.1093","volume":"30","author":[{"given":"Ying","family":"Ding","sequence":"first","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"},{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaowu","family":"Tang","sequence":"additional","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Serena G.","family":"Liao","sequence":"additional","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jia","family":"Jia","sequence":"additional","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Steffi","family":"Oesterreich","sequence":"additional","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yan","family":"Lin","sequence":"additional","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"George C.","family":"Tseng","sequence":"additional","affiliation":[{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"},{"name":"1 Joint Carnegie Mellon University\u2013University of Pittsburgh Ph.D. Program in Computational Biology, 2 Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and 3 Magee-Womens Research Institute, Pittsburgh, PA 15213, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2014,8,1]]},"reference":[{"key":"2023012711595415400_btu520-B1","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1038\/nrg1749","article-title":"Microarray data analysis: from disarray to consolidation and consensus","volume":"7","author":"Allison","year":"2006","journal-title":"Nat. Rev. Genet."},{"key":"2023012711595415400_btu520-B2","article-title":"Correcting the optimally selected resampling-based error rate: a smooth analytical alternative to nested cross-validation","volume-title":"Technical report","author":"Bernau","year":"2011"},{"key":"2023012711595415400_btu520-B3","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1111\/biom.12041","article-title":"Correcting the optimal resampling-based error rate by estimating the error rate of wrapper algorithms","volume":"69","author":"Bernau","year":"2013","journal-title":"Biometrics"},{"key":"2023012711595415400_btu520-B4","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.1093\/bioinformatics\/btl066","article-title":"Avoiding model selection bias in small-sample genomic datasets","volume":"22","author":"Berrar","year":"2006","journal-title":"Bioinformatics"},{"key":"2023012711595415400_btu520-B5","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/1471-2288-9-85","article-title":"Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction","volume":"9","author":"Boulesteix","year":"2009","journal-title":"BMC Med. Res. Methodol."},{"key":"2023012711595415400_btu520-B6","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1038\/nature10983","article-title":"The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups","volume":"486","author":"Curtis","year":"2012","journal-title":"Nature"},{"key":"2023012711595415400_btu520-B7","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1093\/jnci\/djk018","article-title":"Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting","volume":"99","author":"Dupuy","year":"2007","journal-title":"J. Natl Cancer Inst."},{"key":"2023012711595415400_btu520-B8","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1198\/jasa.2009.tm08523","article-title":"Empirical Bayes estimates for large-scale prediction problems","volume":"104","author":"Efron","year":"2009","journal-title":"J. Am. Stat. Assoc."},{"key":"2023012711595415400_btu520-B9","doi-asserted-by":"crossref","first-page":"1979","DOI":"10.1093\/bioinformatics\/bti294","article-title":"Estimating misclassification error with small samples via bootstrap cross-validation","volume":"21","author":"Fu","year":"2005","journal-title":"Bioinformatics"},{"key":"2023012711595415400_btu520-B10","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1089\/106652703321825928","article-title":"Estimating dataset size requirements for classifying DNA microarray data","volume":"10","author":"Mukherjee","year":"2003","journal-title":"J. Comput. Biol."},{"key":"2023012711595415400_btu520-B11","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1186\/1471-2105-9-439","article-title":"CMA: a comprehensive bioconductor package for supervised classification with high dimensional data","volume":"9","author":"Slawski","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2023012711595415400_btu520-B12","doi-asserted-by":"crossref","first-page":"822","DOI":"10.1214\/08-AOAS224","article-title":"A bias correction for the minimum error rate in cross-validation","volume":"3","author":"Tibshirani","year":"2009","journal-title":"Ann. Appl. Stat."},{"key":"2023012711595415400_btu520-B13","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/1471-2105-7-91","article-title":"Bias in error estimation when using cross-validation for model selection","volume":"7","author":"Varma","year":"2006","journal-title":"BMC Bioinformatics"},{"key":"2023012711595415400_btu520-B14","doi-asserted-by":"crossref","first-page":"1363","DOI":"10.1093\/bioinformatics\/btm117","article-title":"Classification based upon gene expression data: bias and precision of error rates","volume":"23","author":"Wood","year":"2007","journal-title":"Bioinformatics"},{"key":"2023012711595415400_btu520-B15","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1093\/bioinformatics\/btp605","article-title":"Reporting bias when using real data sets to analyze classification performance","volume":"26","author":"Yousefi","year":"2010","journal-title":"Bioinformatics"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/30\/22\/3152\/48931071\/bioinformatics_30_22_3152.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/30\/22\/3152\/48931071\/bioinformatics_30_22_3152.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T12:56:14Z","timestamp":1674824174000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/30\/22\/3152\/2390784"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,8,1]]},"references-count":15,"journal-issue":{"issue":"22","published-print":{"date-parts":[[2014,11,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btu520","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2014,11,15]]},"published":{"date-parts":[[2014,8,1]]}}}