{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T17:10:51Z","timestamp":1785517851488,"version":"3.56.0"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"16","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2010,8,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: In statistical bioinformatics research, different optimization mechanisms potentially lead to \u2018over-optimism\u2019 in published papers. So far, however, a systematic critical study concerning the various sources underlying this over-optimism is lacking.<\/jats:p>\n               <jats:p>Results: We present an empirical study on over-optimism using high-dimensional classification as example. Specifically, we consider a \u2018promising\u2019 new classification algorithm, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. While this approach yields poor results in terms of error rate, we quantitatively demonstrate that it can artificially seem superior to existing approaches if we \u2018fish for significance\u2019. The investigated sources of over-optimism include the optimization of datasets, of settings, of competing methods and, most importantly, of the method's characteristics. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should always be demonstrated on independent validation data.<\/jats:p>\n               <jats:p>Availability: The R codes and relevant data can be downloaded from http:\/\/www.ibe.med.uni-muenchen.de\/organisation\/mitarbeiter\/020_professuren\/boulesteix\/overoptimism\/, such that the study is completely reproducible.<\/jats:p>\n               <jats:p>Contact: \u00a0boulesteix@ibe.med.uni-muenchen.de<\/jats:p>","DOI":"10.1093\/bioinformatics\/btq323","type":"journal-article","created":{"date-parts":[[2010,6,28]],"date-time":"2010-06-28T00:12:58Z","timestamp":1277683978000},"page":"1990-1998","source":"Crossref","is-referenced-by-count":84,"title":["Over-optimism in bioinformatics: an illustration"],"prefix":"10.1093","volume":"26","author":[{"given":"Monika","family":"Jelizarow","sequence":"first","affiliation":[{"name":"1 Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, 81377 Munich, Germany, 2SUPELEC Sciences des Syst\u00e8mes (E3S)-Department of Signal Processing and Electronics Systems - 3, rue Joliot Curie, Plateau de Moulon, 91192 Gif-sur-Yvette Cedex, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vincent","family":"Guillemot","sequence":"additional","affiliation":[{"name":"1 Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, 81377 Munich, Germany, 2SUPELEC Sciences des Syst\u00e8mes (E3S)-Department of Signal Processing and Electronics Systems - 3, rue Joliot Curie, Plateau de Moulon, 91192 Gif-sur-Yvette Cedex, France"},{"name":"1 Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, 81377 Munich, Germany, 2SUPELEC Sciences des Syst\u00e8mes (E3S)-Department of Signal Processing and Electronics Systems - 3, rue Joliot Curie, Plateau de Moulon, 91192 Gif-sur-Yvette Cedex, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arthur","family":"Tenenhaus","sequence":"additional","affiliation":[{"name":"1 Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, 81377 Munich, Germany, 2SUPELEC Sciences des Syst\u00e8mes (E3S)-Department of Signal Processing and Electronics Systems - 3, rue Joliot Curie, Plateau de Moulon, 91192 Gif-sur-Yvette Cedex, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Korbinian","family":"Strimmer","sequence":"additional","affiliation":[{"name":"1 Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, 81377 Munich, Germany, 2SUPELEC Sciences des Syst\u00e8mes (E3S)-Department of Signal Processing and Electronics Systems - 3, rue Joliot Curie, Plateau de Moulon, 91192 Gif-sur-Yvette Cedex, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anne-Laure","family":"Boulesteix","sequence":"additional","affiliation":[{"name":"1 Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, 81377 Munich, Germany, 2SUPELEC Sciences des Syst\u00e8mes (E3S)-Department of Signal Processing and Electronics Systems - 3, rue Joliot Curie, Plateau de Moulon, 91192 Gif-sur-Yvette Cedex, France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2010,6,26]]},"reference":[{"key":"2023012508022586500_B1","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1214\/09-AOAS277","article-title":"Feature selection in omics prediction problems using cat scores and false non-discovery rate control","volume":"4","author":"Ahdesm\u00e4ki","year":"2010","journal-title":"Ann. Appl. Stat."},{"key":"2023012508022586500_B2","doi-asserted-by":"crossref","first-page":"6562","DOI":"10.1073\/pnas.102102699","article-title":"Selection bias in gene extraction on the basis of microarray gene-expression data","volume":"99","author":"Ambroise","year":"2002","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012508022586500_B3","doi-asserted-by":"crossref","first-page":"989","DOI":"10.3150\/bj\/1106314847","article-title":"Some theory for Fisher's linear discriminant function, \u201cnaive Bayes\u201d, and some alternatives when there are many more variables than observations","volume":"10","author":"Bickel","year":"2004","journal-title":"Bernoulli"},{"key":"2023012508022586500_B4","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/1471-2105-10-18","article-title":"Incorporating pathway information into boosting estimation of high-dimensional risk prediction models","volume":"10","author":"Binder","year":"2009","journal-title":"BMC Bioinformatics"},{"key":"2023012508022586500_B5","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/1471-2288-9-85","article-title":"Optimal classifier selection and negative bias in error rate estimation: An empirical study on high-dimensional prediction","volume":"9","author":"Boulesteix","year":"2009","journal-title":"BMC Med. Res. Methodol."},{"key":"2023012508022586500_B6","doi-asserted-by":"crossref","first-page":"77","DOI":"10.4137\/CIN.S408","article-title":"Evaluating microarray-based classifiers: an overview","volume":"6","author":"Boulesteix","year":"2008","journal-title":"Cancer Informat."},{"key":"2023012508022586500_B7","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1093\/bioinformatics\/btp648","article-title":"Over-optimism in bioinformatics research","volume":"26","author":"Boulesteix","year":"2010","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B8","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1093\/bioinformatics\/btg419","article-title":"Is cross-validation valid for small-sample microarray classification?","volume":"20","author":"Braga-Neto","year":"2004","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B9","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1109\/MSP.2007.273062","article-title":"Fads and fallacies in the name of small-sample microarray classification","volume":"24","author":"Braga-Neto","year":"2007","journal-title":"IEEE Sign. Process. Mag."},{"key":"2023012508022586500_B10","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/1471-2288-8-18","article-title":"Reducing the probability of false positive research findings by pre-publication validation: Experience with a large multiple sclerosis database","volume":"8","author":"Daumer","year":"2008","journal-title":"BMC Med. Res. Methodol."},{"key":"2023012508022586500_B11","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1023\/A:1007413511361","article-title":"On the optimality of the simple Bayesian classifier under zero-one loss","volume":"29","author":"Domingos","year":"1997","journal-title":"Mach. Learn."},{"key":"2023012508022586500_B12","doi-asserted-by":"crossref","first-page":"69","DOI":"10.2174\/138920208784139546","article-title":"On the epistemological crisis in genomics","volume":"9","author":"Dougherty","year":"2008","journal-title":"Curr. Genomics"},{"key":"2023012508022586500_B13","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1198\/016214502753479248","article-title":"Comparison of discrimination methods for the classification of tumors using gene expression data","volume":"97","author":"Dudoit","year":"2002","journal-title":"J. Am. Stat. Assoc."},{"key":"2023012508022586500_B14","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1038\/scientificamerican0577-119","article-title":"Stein's paradox in statistics","volume":"236","author":"Efron","year":"1977","journal-title":"Sci. Am."},{"key":"2023012508022586500_B15","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1080\/01621459.1989.10478752","article-title":"Regularized discriminant analysis","volume":"84","author":"Friedman","year":"1989","journal-title":"J. Am. Stat. Assoc."},{"key":"2023012508022586500_B16","author":"Golub","year":"2010","journal-title":"golubEsets."},{"key":"2023012508022586500_B17","first-page":"207","article-title":"Graph-Constrained Discriminant Analysis of functional genomics data","volume-title":"IEEE International Conference on Bioinformatics and Biomedicine Worshops","author":"Guillemot","year":"2008"},{"key":"2023012508022586500_B18","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1093\/biostatistics\/kxj035","article-title":"Regularized linear discriminant analysis and its application in microarrays","volume":"8","author":"Guo","year":"2007","journal-title":"Biostatistics"},{"key":"2023012508022586500_B19","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1093\/biomet\/asp081","article-title":"Incorporating prior probabilities into high-dimensional classifiers","volume":"97","author":"Hall","year":"2010","journal-title":"Biometrika"},{"key":"2023012508022586500_B20","first-page":"38473","article-title":"Decorrelation of the true and estimated classifier errors in high-dimensional settings","volume":"207","author":"Hanczar","year":"2007","journal-title":"EURASIP J. Bioinformatics Syst. Biol."},{"key":"2023012508022586500_B21","doi-asserted-by":"crossref","first-page":"822","DOI":"10.1093\/bioinformatics\/btq037","article-title":"Small-sample precision of roc-related estimates","volume":"26","author":"Hanczar","year":"2010","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B22","first-page":"1","article-title":"Classifier technology and the illusion of progress","volume":"21","author":"Hand","year":"2006","journal-title":"Stat. Sci."},{"key":"2023012508022586500_B23","first-page":"1469","article-title":"Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks","volume":"10","author":"Hausser","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"2023012508022586500_B24","first-page":"433","article-title":"Group Lasso with Overlap and Graph Lasso","volume-title":"International Conference on Machine Learning (ICML 26).","author":"Jacob","year":"2009"},{"key":"2023012508022586500_B25","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1093\/nar\/28.1.27","article-title":"KEGG: Kyoto Encyclopedia of Genes and Genomes","volume":"28","author":"Kanehisa","year":"2000","journal-title":"Nucleic Acids Res."},{"key":"2023012508022586500_B26","doi-asserted-by":"crossref","first-page":"376","DOI":"10.1093\/biostatistics\/kxq033","article-title":"Reproducible research and the substantive context","volume":"11","author":"Keiding","year":"2010","journal-title":"Biostatistics"},{"key":"2023012508022586500_B27","doi-asserted-by":"crossref","first-page":"1060","DOI":"10.1002\/bies.20294","article-title":"Revisiting scale-free networks","volume":"27","author":"Keller","year":"2005","journal-title":"BioEssays"},{"key":"2023012508022586500_B28","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1016\/S0927-5398(03)00007-0","article-title":"Improved estimation of the covariance matrix of stock returns with an application to portfolio selection","volume":"10","author":"Ledoit","year":"2003","journal-title":"J. Empir. Finan."},{"key":"2023012508022586500_B29","doi-asserted-by":"crossref","first-page":"110","DOI":"10.3905\/jpm.2004.110","article-title":"Honey, I shrunk the sample covariance matrix","volume":"31","author":"Ledoit","year":"2004","journal-title":"J. Portf. Manag."},{"key":"2023012508022586500_B30","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1093\/bioinformatics\/btm201","article-title":"Comparative analysis of microarray normalization procedures: effects on reverse engineering gene networks","volume":"23","author":"Lim","year":"2007","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B31","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1093\/bioinformatics\/btn081","article-title":"Network-constrained regularization and variable selection for analysis of genomic data","volume":"24","author":"Li","year":"2008","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B32","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/ng1422","article-title":"Towards sound epistemological foundations of statistical methods for high-dimensional biology","volume":"36","author":"Mehta","year":"2004","journal-title":"Nat. Genet."},{"key":"2023012508022586500_B33","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1017\/S0305004100030401","article-title":"A generalized inverse for matrices","volume":"51","author":"Penrose","year":"1955","journal-title":"Proc. Camb. Philo. Soc."},{"key":"2023012508022586500_B34","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1186\/1471-2105-8-35","article-title":"Classification of microarray data using gene networks","volume":"8","author":"Rapaport","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023012508022586500_B35","doi-asserted-by":"crossref","first-page":"701","DOI":"10.1093\/bioinformatics\/btp038","article-title":"Papers on normalization, variable selection, classification or clustering of microarray data","volume":"25","author":"Rocke","year":"2009","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B36","doi-asserted-by":"crossref","DOI":"10.2202\/1544-6115.1175","article-title":"A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics","volume":"4","author":"Sch\u00e4fer","year":"2005","journal-title":"Stat. Appl. Genet. Mol. Biol."},{"key":"2023012508022586500_B37","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1016\/S1535-6108(02)00030-2","article-title":"Gene expression correlates of clinical prostate cancer behavior","volume":"1","author":"Singh","year":"2002","journal-title":"Cancer Cell"},{"key":"2023012508022586500_B38","doi-asserted-by":"crossref","DOI":"10.1214\/09-AOAS302","article-title":"Feature selection guided by structural information","volume":"4","author":"Slawski","year":"2010","journal-title":"Ann. Appl. Stat."},{"key":"2023012508022586500_B39","doi-asserted-by":"crossref","first-page":"3","DOI":"10.2202\/1544-6115.1027","article-title":"Linear models and empirical Bayes methods for assessing differential expression in microarray experiments","volume":"3","author":"Smyth","year":"2004","journal-title":"Stat. Appl. Genet. Mol. Biol."},{"key":"2023012508022586500_B40","first-page":"197","article-title":"Inadmissibility of the usual estimator for the mean of a multivariate normal distribution","volume-title":"Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability.","author":"Stein","year":"1955"},{"key":"2023012508022586500_B41","doi-asserted-by":"crossref","first-page":"3170","DOI":"10.1093\/bioinformatics\/btm488","article-title":"Incorporating prior knowledge of gene functional groups into regularized discriminant analysis of microarray data","volume":"23","author":"Tai","year":"2007","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B42","doi-asserted-by":"crossref","first-page":"1775","DOI":"10.1093\/bioinformatics\/btm234","article-title":"Incorporating prior knowledge of predictors into penalized classifiers with multiple penalty terms","volume":"23","author":"Tai","year":"2007","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B43","doi-asserted-by":"crossref","first-page":"6567","DOI":"10.1073\/pnas.082099299","article-title":"Diagnosis of multiple cancer types by shrunken centroids of gene expression","volume":"99","author":"Tibshirani","year":"2002","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012508022586500_B44","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1016\/S0140-6736(05)17947-1","article-title":"Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer","volume":"365","author":"Wang","year":"2005","journal-title":"Lancet"},{"key":"2023012508022586500_B45","author":"Whalen","year":"2010","journal-title":"CLL."},{"key":"2023012508022586500_B46","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1093\/bioinformatics\/btp605","article-title":"Reporting bias when using real datasets to analyze classification performance","volume":"26","author":"Yousefi","year":"2010","journal-title":"Bioinformatics"},{"key":"2023012508022586500_B47","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1186\/1471-2105-10-337","article-title":"Classification and biomarker identification using gene network modules and support vector machines","volume":"10","author":"Yousef","year":"2009","journal-title":"BMC Bioinformatics"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/26\/16\/1990\/48854623\/bioinformatics_26_16_1990.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/26\/16\/1990\/48854623\/bioinformatics_26_16_1990.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,25]],"date-time":"2023-01-25T08:03:43Z","timestamp":1674633823000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/26\/16\/1990\/216600"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,6,26]]},"references-count":47,"journal-issue":{"issue":"16","published-print":{"date-parts":[[2010,8,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btq323","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2010,8,15]]},"published":{"date-parts":[[2010,6,26]]}}}