{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T05:37:11Z","timestamp":1772084231980,"version":"3.50.1"},"reference-count":29,"publisher":"Oxford University Press (OUP)","issue":"22","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2013,11,15]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Motivation: After more than a decade since microarrays were used to predict phenotype of biological samples, real-life applications for disease screening and identification of patients who would best benefit from treatment are still emerging. The interest of the scientific community in identifying best approaches to develop such prediction models was reaffirmed in a competition style international collaboration called IMPROVER Diagnostic Signature Challenge whose results we describe herein.<\/jats:p><jats:p>Results: Fifty-four teams used public data to develop prediction models in four disease areas including multiple sclerosis, lung cancer, psoriasis and chronic obstructive pulmonary disease, and made predictions on blinded new data that we generated. Teams were scored using three metrics that captured various aspects of the quality of predictions, and best performers were awarded. This article presents the challenge results and introduces to the community the approaches of the best overall three performers, as well as an R package that implements the approach of the best overall team.<\/jats:p><jats:p>The analyses of model performance data submitted in the challenge as well as additional simulations that we have performed revealed that (i) the quality of predictions depends more on the disease endpoint than on the particular approaches used in the challenge; (ii) the most important modeling factor (e.g. data preprocessing, feature selection and classifier type) is problem dependent; and (iii) for optimal results datasets and methods have to be carefully matched. Biomedical factors such as the disease severity and confidence in diagnostic were found to be associated with the misclassification rates across the different teams.<\/jats:p><jats:p>Availability: The lung cancer dataset is available from Gene Expression Omnibus (accession, GSE43580). The maPredictDSC R package implementing the approach of the best overall team is available at www.bioconductor.org or http:\/\/bioinformaticsprb.med.wayne.edu\/.<\/jats:p><jats:p>Contact: \u00a0gustavo@us.ibm.com<\/jats:p><jats:p>Supplementary information: \u00a0Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btt492","type":"journal-article","created":{"date-parts":[[2013,8,22]],"date-time":"2013-08-22T00:14:38Z","timestamp":1377130478000},"page":"2892-2899","source":"Crossref","is-referenced-by-count":103,"title":["Strengths and limitations of microarray-based phenotype prediction: lessons learned from the IMPROVER Diagnostic Signature Challenge"],"prefix":"10.1093","volume":"29","author":[{"given":"Adi L.","family":"Tarca","sequence":"first","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"},{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Mario","family":"Lauria","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Michael","family":"Unger","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Erhan","family":"Bilal","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Stephanie","family":"Boue","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Kushal","family":"Kumar Dey","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Julia","family":"Hoeng","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Heinz","family":"Koeppl","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Florian","family":"Martin","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Pablo","family":"Meyer","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Preetam","family":"Nandy","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Raquel","family":"Norel","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Manuel","family":"Peitsch","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Jeremy J.","family":"Rice","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Roberto","family":"Romero","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Gustavo","family":"Stolovitzky","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Marja","family":"Talikka","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Yang","family":"Xiang","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"given":"Christoph","family":"Zechner","sequence":"additional","affiliation":[{"name":"1 Department of Computer Science, Wayne State University, 2Perinatology Research Branch, NICHD\/NIH, Detroit, MI 48201, USA, 3The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, 4ETH Zurich, Zurich 8092, Switzerland, 5IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and 6Philip Morris International, Research & Development, Neuch\u00e2tel CH-2000, Switzerland"}]},{"name":"IMPROVER DSC Collaborators","sequence":"additional","affiliation":[]}],"member":"286","published-online":{"date-parts":[[2013,8,20]]},"reference":[{"key":"2023012810465748000_btt492-B1","first-page":"789","article-title":"(2006) NSABP study confirms oncotype DX predicts chemotherapy benefit in breast cancer patients","volume":"20","year":"2006","journal-title":"Oncology (Williston Park)"},{"key":"2023012810465748000_btt492-B2","author":"Acharya","year":"2012"},{"key":"2023012810465748000_btt492-B3","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1093\/bioinformatics\/16.5.412","article-title":"Assessing the accuracy of prediction algorithms for classification: an overview","volume":"16","author":"Baldi","year":"2000","journal-title":"Bioinformatics"},{"key":"2023012810465748000_btt492-B4","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"J. Royal Stat. Soc. B"},{"key":"2023012810465748000_btt492-B5","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1038\/35020115","article-title":"Molecular classification of cutaneous malignant melanoma by gene expression profiling","volume":"406","author":"Bittner","year":"2000","journal-title":"Nature"},{"key":"2023012810465748000_btt492-B6","first-page":"4963","article-title":"Translation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma","volume":"62","author":"Gordon","year":"2002","journal-title":"Cancer Res."},{"key":"2023012810465748000_btt492-B25","doi-asserted-by":"crossref","first-page":"201","DOI":"10.3390\/jpm2040201","article-title":"Insurance Coverage Policies for Personalized Medicine","volume":"2","author":"Hresko","year":"2012","journal-title":"J. Pers. Med."},{"key":"2023012810465748000_btt492-B7","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1093\/biostatistics\/4.2.249","article-title":"Exploration, normalization, and summaries of high density oligonucleotide array probe level data","volume":"4","author":"Irizarry","year":"2003","journal-title":"Biostatistics"},{"key":"2023012810465748000_btt492-B8","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1586\/14737159.6.6.803","article-title":"A genetic signature can predict prognosis and response to therapy in breast cancer: Oncotype DX","volume":"6","author":"Kaklamani","year":"2006","journal-title":"Expert. Rev. Mol. Diagn."},{"key":"2023012810465748000_btt492-B9","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/j.jaad.2004.04.012","article-title":"Evaluating psoriasis with Psoriasis Area and Severity Index, Psoriasis Global Assessment, and Lattice System Physician's Global Assessment","volume":"51","author":"Langley","year":"2004","journal-title":"J. Am. Acad. Dermatol."},{"key":"2023012810465748000_btt492-B26","doi-asserted-by":"crossref","DOI":"10.4161\/sysb.25982","article-title":"Rank-based transcriptional signatures: a novel approach to diagnostic biomarker definition and analysis","author":"Lauria","year":"2013","journal-title":"Syst. Biomed."},{"key":"2023012810465748000_btt492-B10","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1038\/tpj.2010.57","article-title":"A comparison of batch effect removal methods for enhancement of prediction performance using MAQC-II microarray gene expression data","volume":"10","author":"Luo","year":"2010","journal-title":"Pharmacogenomics J."},{"key":"2023012810465748000_btt492-B11","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1038\/nmeth.2016","article-title":"Wisdom of crowds for robust gene network inference","volume":"9","author":"Marbach","year":"2012","journal-title":"Nat. Methods"},{"key":"2023012810465748000_btt492-B12","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","article-title":"Comparison of the predicted and observed secondary structure of T4 phage lysozyme","volume":"405","author":"Matthews","year":"1975","journal-title":"Biochim. Biophys. Acta"},{"key":"2023012810465748000_btt492-B13","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1038\/nbt.1968","article-title":"Verification of systems biology research in the age of collaborative competition","volume":"29","author":"Meyer","year":"2011","journal-title":"Nat. Biotechnol."},{"key":"2023012810465748000_btt492-B14","doi-asserted-by":"crossref","first-page":"1193","DOI":"10.1093\/bioinformatics\/bts116","article-title":"Industrial methodology for process verification in research (IMPROVER): toward systems biology verification","volume":"28","author":"Meyer","year":"2012","journal-title":"Bioinformatics"},{"key":"2023012810465748000_btt492-B15","first-page":"147","article-title":"Individualization of therapy using MammaPrint: from development to the MINDACT Trial","volume":"4","author":"Mook","year":"2007","journal-title":"Cancer Genomics Proteomics"},{"key":"2023012810465748000_btt492-B29","doi-asserted-by":"crossref","DOI":"10.4161\/sysb.25271","article-title":"Learning diagnostic signatures from microarray data using L1-regularized logistic regression","author":"Nandy","year":"2013","journal-title":"Syst. Biomed."},{"key":"2023012810465748000_btt492-B16","doi-asserted-by":"crossref","first-page":"3257","DOI":"10.1245\/s10434-012-2561-6","article-title":"Comparison of molecular subtyping with BluePrint, MammaPrint, and TargetPrint to local clinical subtyping in breast cancer patients","volume":"19","author":"Nguyen","year":"2012","journal-title":"Ann. Surg. Oncol."},{"key":"2023012810465748000_btt492-B17","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1038\/35021093","article-title":"Molecular portraits of human breast tumours","volume":"406","author":"Perou","year":"2000","journal-title":"Nature"},{"key":"2023012810465748000_btt492-B18","doi-asserted-by":"crossref","first-page":"e9202","DOI":"10.1371\/journal.pone.0009202","article-title":"Towards a rigorous assessment of systems biology models: the DREAM3 challenges","volume":"5","author":"Prill","year":"2010","journal-title":"PLoS One"},{"key":"2023012810465748000_btt492-B19","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1126\/science.270.5235.467","article-title":"Quantitative monitoring of gene expression patterns with a complementary DNA microarray","volume":"270","author":"Schena","year":"1995","journal-title":"Science"},{"key":"2023012810465748000_btt492-B20","doi-asserted-by":"crossref","first-page":"2498","DOI":"10.1101\/gr.1239303","article-title":"Cytoscape: a software environment for integrated models of biomolecular interaction networks","volume":"13","author":"Shannon","year":"2003","journal-title":"Genome Res."},{"key":"2023012810465748000_btt492-B21","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1038\/nbt.1665","article-title":"The MicroArray Quality Control (MAQC)-II study of common practices for the development and validation of microarray-based predictive models","volume":"28","author":"Shi","year":"2010","journal-title":"Nat. Biotechnol."},{"key":"2023012810465748000_btt492-B22","doi-asserted-by":"crossref","first-page":"e116","DOI":"10.1371\/journal.pcbi.0030116","article-title":"Machine learning and its applications to biology","volume":"3","author":"Tarca","year":"2007","journal-title":"PLoS. Comput. Biol."},{"key":"2023012810465748000_btt492-B27","doi-asserted-by":"crossref","DOI":"10.4161\/sysb.25980","article-title":"Methodological Approach from the Best Overall Team in the IMPROVER Diagnostic Signature Challenge","author":"Tarca","year":"2013","journal-title":"Syst. Biomed."},{"key":"2023012810465748000_btt492-B28","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. Royal. Statist. Soc B."},{"key":"2023012810465748000_btt492-B23","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1111\/j.1399-0012.2007.00681.x","article-title":"Post-transplant ischemic injury is associated with up-regulated AlloMap gene expression","volume":"21","author":"Yamani","year":"2007","journal-title":"Clin. Transplant."},{"key":"2023012810465748000_btt492-B24","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/j.healun.2006.12.011","article-title":"Transplant vasculopathy is associated with increased AlloMap gene expression score","volume":"26","author":"Yamani","year":"2007","journal-title":"J. Heart Lung Transplant."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/29\/22\/2892\/48892400\/bioinformatics_29_22_2892.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/29\/22\/2892\/48892400\/bioinformatics_29_22_2892.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:01:54Z","timestamp":1715904114000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/29\/22\/2892\/313807"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,8,20]]},"references-count":29,"journal-issue":{"issue":"22","published-print":{"date-parts":[[2013,11,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btt492","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2013,11,15]]},"published":{"date-parts":[[2013,8,20]]}}}