{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T09:56:13Z","timestamp":1760608573043},"reference-count":20,"publisher":"Oxford University Press (OUP)","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2010,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: It is commonplace for authors to propose a new classification rule, either the operator construction part or feature selection, and demonstrate its performance on real data sets, which often come from high-dimensional studies, such as from gene-expression microarrays, with small samples. Owing to the variability in feature selection and error estimation, individual reported performances are highly imprecise. Hence, if only the best test results are reported, then these will be biased relative to the overall performance of the proposed procedure.<\/jats:p>\n               <jats:p>Results: This article characterizes reporting bias with several statistics and computes these statistics in a large simulation study using both modeled and real data. The results appear as curves giving the different reporting biases as functions of the number of samples tested when reporting only the best or second best performance. It does this for two classification rules, linear discriminant analysis (LDA) and 3-nearest-neighbor (3NN), and for filter and wrapper feature selection, t-test and sequential forward search. These were chosen on account of their well-studied properties and because they were amenable to the extremely large amount of processing required for the simulations. The results across all the experiments are consistent: there is generally large bias overriding what would be considered a significant performance differential, when reporting the best or second best performing data set. We conclude that there needs to be a database of data sets and that, for those studies depending on real data, results should be reported for all data sets in the database.<\/jats:p>\n               <jats:p>Availability: Companion web site at http:\/\/gsp.tamu.edu\/Publications\/supplementary\/yousefi09a\/<\/jats:p>\n               <jats:p>Contact: \u00a0edward@ece.tamu.edu<\/jats:p>\n               <jats:p>Supplementary information: Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btp605","type":"journal-article","created":{"date-parts":[[2009,10,22]],"date-time":"2009-10-22T00:34:28Z","timestamp":1256171668000},"page":"68-76","source":"Crossref","is-referenced-by-count":43,"title":["Reporting bias when using real data sets to analyze classification performance"],"prefix":"10.1093","volume":"26","author":[{"given":"Mohammadmahdi R.","family":"Yousefi","sequence":"first","affiliation":[{"name":"1 Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843 and 2 Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ 85004, USA"}]},{"given":"Jianping","family":"Hua","sequence":"additional","affiliation":[{"name":"1 Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843 and 2 Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ 85004, USA"}]},{"given":"Chao","family":"Sima","sequence":"additional","affiliation":[{"name":"1 Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843 and 2 Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ 85004, USA"}]},{"given":"Edward R.","family":"Dougherty","sequence":"additional","affiliation":[{"name":"1 Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843 and 2 Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ 85004, USA"},{"name":"1 Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843 and 2 Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ 85004, USA"}]}],"member":"286","published-online":{"date-parts":[[2009,10,21]]},"reference":[{"key":"2023012507525983900_B1","doi-asserted-by":"crossref","first-page":"13790","DOI":"10.1073\/pnas.191502998","article-title":"Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses","volume":"98","author":"Bhattacharjee","year":"2001","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023012507525983900_B2","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1093\/bioinformatics\/btg419","article-title":"Is cross-validation valid for small-sample microarray classification?","volume":"20","author":"Braga-Neto","year":"2004","journal-title":"Bioinformatics"},{"key":"2023012507525983900_B3","doi-asserted-by":"crossref","first-page":"1198","DOI":"10.1038\/modpathol.3800167","article-title":"Novel endothelial cell markers in hepatocellular carcinoma","volume":"17","author":"Chen","year":"2004","journal-title":"Modern Pathol."},{"key":"2023012507525983900_B4","doi-asserted-by":"crossref","first-page":"3207","DOI":"10.1158\/1078-0432.CCR-06-2765","article-title":"Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the TRANSBIG multicenter independent validation series","volume":"13","author":"Desmedt","year":"2007","journal-title":"Clin. Cancer Res."},{"key":"2023012507525983900_B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2174\/138920207780076956","article-title":"Validation of computational methods in genomics","volume":"8","author":"Dougherty","year":"2007","journal-title":"Curr. Genomics"},{"key":"2023012507525983900_B6","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/0031-3203(78)90029-8","article-title":"Additive estimators for probabilities of correct classification","volume":"10","author":"Glick","year":"1978","journal-title":"Pattern Recogn."},{"key":"2023012507525983900_B7","doi-asserted-by":"crossref","DOI":"10.1155\/2007\/38473","article-title":"Decorrelation of the true and estimated classifier errors in high-dimensional settings","author":"Hanczar","year":"2007","journal-title":"EURASIP J. Bioinform. Syst. Biol."},{"key":"2023012507525983900_B8","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1016\/j.patcog.2008.08.001","article-title":"Performance of feature selection methods in the classification of high-dimensional data","volume":"42","author":"Hua","year":"2009","journal-title":"Pattern Recogn."},{"key":"2023012507525983900_B9","doi-asserted-by":"crossref","first-page":"724","DOI":"10.1101\/gr.2807605","article-title":"Classification of a large microarray data set: algorithm comparison and analysis of drug signatures","volume":"15","author":"Natsoulis","year":"2005","journal-title":"Genome Res."},{"key":"2023012507525983900_B10","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1056\/NEJMoa060467","article-title":"A genomic strategy to refine prognosis in early-stage non-small-cell lung cancer","volume":"355","author":"Potti","year":"2006","journal-title":"N. Eng. J. Med."},{"key":"2023012507525983900_B11","doi-asserted-by":"crossref","first-page":"1937","DOI":"10.1056\/NEJMoa012914","article-title":"The use of molecular profiling to predict survival after chemotherapy for diffuse large-B-cell lymphoma","volume":"346","author":"Rosenwald","year":"2002","journal-title":"N. Eng. J. Med."},{"key":"2023012507525983900_B12","doi-asserted-by":"crossref","DOI":"10.1515\/9781400865260","volume-title":"Genomic Signal Processing.","author":"Shmulevich","year":"2007"},{"key":"2023012507525983900_B13","doi-asserted-by":"crossref","first-page":"2430","DOI":"10.1093\/bioinformatics\/btl407","article-title":"What should be expected from feature selection in small-sample settings","volume":"22","author":"Sima","year":"2006","journal-title":"Bioinformatics"},{"key":"2023012507525983900_B14","doi-asserted-by":"crossref","first-page":"2472","DOI":"10.1016\/j.patcog.2005.03.026","article-title":"Impact of error estimation on feature-selection algorithms","volume":"38","author":"Sima","year":"2005","journal-title":"Pattern Recogn."},{"key":"2023012507525983900_B15","first-page":"7388","article-title":"Molecular classification of human carcinomas by use of gene expression signatures","volume":"61","author":"Su","year":"2001","journal-title":"Cancer Res."},{"key":"2023012507525983900_B16","doi-asserted-by":"crossref","first-page":"1617","DOI":"10.1056\/NEJMoa040465","article-title":"Prognostically useful gene-expression profiles in acute myeloid leukemia","volume":"350","author":"Valk","year":"2004","journal-title":"N. Eng. J. Med."},{"key":"2023012507525983900_B17","doi-asserted-by":"crossref","first-page":"1999","DOI":"10.1056\/NEJMoa021967","article-title":"A gene-expression signature as a predictor of survival in breast cancer","volume":"347","author":"van de Vijver","year":"2002","journal-title":"N. Eng. J. Med."},{"key":"2023012507525983900_B18","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1016\/S0140-6736(05)17947-1","article-title":"Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer","volume":"365","author":"Wang","year":"2005","journal-title":"Lancet"},{"key":"2023012507525983900_B19","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1016\/S1535-6108(02)00032-6","article-title":"Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling","volume":"1","author":"Yeoh","year":"2002","journal-title":"Cancer Cell"},{"key":"2023012507525983900_B20","doi-asserted-by":"crossref","first-page":"2020","DOI":"10.1182\/blood-2005-11-013458","article-title":"The molecular classification of multiple myeloma","volume":"108","author":"Zhan","year":"2006","journal-title":"Blood"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/26\/1\/68\/48851537\/bioinformatics_26_1_68.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/26\/1\/68\/48851537\/bioinformatics_26_1_68.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,25]],"date-time":"2023-01-25T07:53:22Z","timestamp":1674633202000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/26\/1\/68\/181625"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,10,21]]},"references-count":20,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btp605","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2010,1,1]]},"published":{"date-parts":[[2009,10,21]]}}}