{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T19:42:31Z","timestamp":1775072551006,"version":"3.50.1"},"reference-count":26,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T00:00:00Z","timestamp":1580083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["GM121312"],"award-info":[{"award-number":["GM121312"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["GM103456"],"award-info":[{"award-number":["GM103456"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"William K. Warren Jr. Foundation"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Summary<\/jats:title>\n                    <jats:p>Feature selection can improve the accuracy of machine-learning models, but appropriate steps must be taken to avoid overfitting. Nested cross-validation (nCV) is a common approach that chooses the classification model and features to represent a given outer fold based on features that give the maximum inner-fold accuracy. Differential privacy is a related technique to avoid overfitting that uses a privacy-preserving noise mechanism to identify features that are stable between training and holdout sets.<\/jats:p>\n                    <jats:p>We develop consensus nested cross-validation (cnCV) that combines the idea of feature stability from differential privacy with nCV. Feature selection is applied in each inner fold and the consensus of top features across folds is used as a measure of feature stability or reliability instead of classification accuracy, which is used in standard nCV. We use simulated data with main effects, correlation and interactions to compare the classification accuracy and feature selection performance of the new cnCV with standard nCV, Elastic Net optimized by cross-validation, differential privacy and private evaporative cooling (pEC). We also compare these methods using real RNA-seq data from a study of major depressive disorder.<\/jats:p>\n                    <jats:p>The cnCV method has similar training and validation accuracy to nCV, but cnCV has much shorter run times because it does not construct classifiers in the inner folds. The cnCV method chooses a more parsimonious set of features with fewer false positives than nCV. The cnCV method has similar accuracy to pEC and cnCV selects stable features between folds without the need to specify a privacy threshold. We show that cnCV is an effective and efficient approach for combining feature selection with classification.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Code available at https:\/\/github.com\/insilico\/cncv.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btaa046","type":"journal-article","created":{"date-parts":[[2020,1,20]],"date-time":"2020-01-20T15:10:46Z","timestamp":1579533046000},"page":"3093-3098","source":"Crossref","is-referenced-by-count":184,"title":["Consensus features nested cross-validation"],"prefix":"10.1093","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0917-2784","authenticated-orcid":false,"given":"Saeid","family":"Parvandeh","sequence":"first","affiliation":[{"name":"Tandy School of Computer Science , University of Tulsa, Tulsa, OK, USA"},{"name":"Department of Molecular and Human Genetics , Baylor College of Medicine, Houston, TX, USA"}]},{"given":"Hung-Wen","family":"Yeh","sequence":"additional","affiliation":[{"name":"Health Services and Outcomes Research , Children\u2019s Mercy Hospital, Kansas City, MO, USA"}]},{"given":"Martin P","family":"Paulus","sequence":"additional","affiliation":[{"name":"Laureate Institute for Brain Research , Tulsa, OK, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9494-8833","authenticated-orcid":false,"given":"Brett A","family":"McKinney","sequence":"additional","affiliation":[{"name":"Tandy School of Computer Science , University of Tulsa, Tulsa, OK, USA"},{"name":"Department of Mathematics , University of Tulsa, Tulsa, OK, USA"}]}],"member":"286","published-online":{"date-parts":[[2020,1,27]]},"reference":[{"key":"2023013112044752500_btaa046-B1","first-page":"1089","article-title":"No unbiased estimator of the variance of K-fold cross-validation","volume":"5","author":"Bengio","year":"2003","journal-title":"J. Mach. Learn. Res"},{"key":"2023013112044752500_btaa046-B2","first-page":"2079","article-title":"On over-fitting in model selection and subsequent selection bias in performance evaluation","volume":"11","author":"Cawley","year":"2010","journal-title":"J. Mach. Learn. Res"},{"key":"2023013112044752500_btaa046-B3","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1126\/science.aaa9375","article-title":"STATISTICS. The reusable holdout: preserving validity in adaptive data analysis","volume":"349","author":"Dwork","year":"2015","journal-title":"Science"},{"key":"2023013112044752500_btaa046-B4","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1561\/0400000042","article-title":"The algorithmic foundations of differential privacy","volume":"9","author":"Dwork","year":"2013","journal-title":"Found. Trends Theor. Comput. Sci"},{"key":"2023013112044752500_btaa046-B5","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1002\/cem.1225","article-title":"Repeated double cross validation","volume":"23","author":"Filzmoser","year":"2009","journal-title":"J. Chemom"},{"key":"2023013112044752500_btaa046-B6","first-page":"1157","article-title":"An introduction to variable and feature selection","volume":"3","author":"Guyon","year":"2003","journal-title":"J. Mach. Learn. Res"},{"key":"2023013112044752500_btaa046-B7","first-page":"1137","author":"Kohavi","year":"1995"},{"key":"2023013112044752500_btaa046-B8","first-page":"171","volume-title":"Estimating attributes: analysis and extensions of RELIEF.","author":"Kononenko","year":"1994"},{"key":"2023013112044752500_btaa046-B9","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/s13040-015-0040-x","article-title":"Differential co-expression network centrality and machine learning feature selection for identifying susceptibility hubs in networks with scale-free structure","volume":"8","author":"Lareau","year":"2015","journal-title":"BioData Min"},{"key":"2023013112044752500_btaa046-B10","doi-asserted-by":"crossref","first-page":"2906","DOI":"10.1093\/bioinformatics\/btx298","article-title":"Differential privacy-based evaporative cooling feature selection and classification with relief-F and random forests","volume":"33","author":"Le","year":"2017","journal-title":"Bioinformatics"},{"key":"2023013112044752500_btaa046-B11","article-title":"Nearest-neighbor Projected-Distance Regression (NPDR) detects network interactions and controls for confounding and multiple testing","author":"Le","year":"2019","journal-title":"Bioinformatics"},{"key":"2023013112044752500_btaa046-B12","doi-asserted-by":"crossref","first-page":"1358","DOI":"10.1093\/bioinformatics\/bty788","article-title":"STatistical Inference Relief (STIR) feature selection","volume":"35","author":"Le","year":"2019","journal-title":"Bioinformatics"},{"key":"2023013112044752500_btaa046-B13","doi-asserted-by":"crossref","first-page":"3301","DOI":"10.1093\/bioinformatics\/bti499","article-title":"Prediction error estimation: a comparison of resampling methods","volume":"21","author":"Molinaro","year":"2005","journal-title":"Bioinformatics"},{"key":"2023013112044752500_btaa046-B14","doi-asserted-by":"crossref","first-page":"1267","DOI":"10.1038\/mp.2013.161","article-title":"Type I interferon signaling genes in recurrent major depression: increased expression detected by whole-blood RNA sequencing HHS public access","volume":"19","author":"Mostafavi","year":"2014","journal-title":"Mol. Psychiatry"},{"key":"2023013112044752500_btaa046-B15","doi-asserted-by":"crossref","first-page":"2329","DOI":"10.1093\/bioinformatics\/bty965","article-title":"EpistasisRank and EpistasisKatz: interaction network centrality methods that integrate prior knowledge networks","volume":"35","author":"Parvandeh","year":"2019","journal-title":"Bioinformatics"},{"key":"2023013112044752500_btaa046-B16","doi-asserted-by":"crossref","first-page":"79","DOI":"10.3390\/microorganisms7030079","article-title":"Multi-level model to predict antibody response to influenza vaccine using gene expression interaction network feature selection","volume":"7","author":"Parvandeh","year":"2019","journal-title":"Microorganisms"},{"key":"2023013112044752500_btaa046-B17","doi-asserted-by":"crossref","first-page":"D833","DOI":"10.1093\/nar\/gkw943","article-title":"DisGeNET: a comprehensive platform integrating information on human disease-associated genes and variants","volume":"45","author":"Pi\u00f1ero","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2023013112044752500_btaa046-B18","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1093\/jnci\/95.1.14","article-title":"Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification","volume":"95","author":"Simon","year":"2003","journal-title":"J. Natl. Cancer Inst"},{"key":"2023013112044752500_btaa046-B19","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","article-title":"Cross-validatory choice and assessment of statistical predictions","volume":"36","author":"Stone","year":"1974","journal-title":"J. R. Stat. Soc. Series B Methodol"},{"key":"2023013112044752500_btaa046-B20","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1002\/(SICI)1097-0258(19970228)16:4<385::AID-SIM380>3.0.CO;2-3","article-title":"The lasso method for variable selection in the Cox model","volume":"16","author":"Tibshirani","year":"1997","journal-title":"Stat. Med"},{"key":"2023013112044752500_btaa046-B21","first-page":"pp. 1","author":"Tsamardinos","year":"2014"},{"key":"2023013112044752500_btaa046-B22","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/j.jbi.2018.07.014","article-title":"Relief-based feature selection: introduction and review","volume":"85","author":"Urbanowicz","year":"2018","journal-title":"J. Biomed. Inform"},{"key":"2023013112044752500_btaa046-B23","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/1471-2105-7-91","article-title":"Bias in error estimation when using cross-validation for model selection","volume":"7","author":"Varma","year":"2006","journal-title":"BMC Bioinformatics"},{"key":"2023013112044752500_btaa046-B24","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.neuroimage.2016.10.038","article-title":"Assessing and tuning brain decoders: cross-validation, caveats, and guidelines","volume":"145","author":"Varoquaux","year":"2017","journal-title":"Neuroimage"},{"key":"2023013112044752500_btaa046-B25","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1111\/adb.12644","article-title":"Classifying and characterizing nicotine use disorder with high accuracy using machine learning and resting-state FMRI","volume":"24","author":"Wetherill","year":"2019","journal-title":"Addict. Biol"},{"key":"2023013112044752500_btaa046-B26","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"Zou","year":"2005","journal-title":"J. R. Stat. Soc. Series B Stat. Methodol"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btaa046\/32980730\/btaa046.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/10\/3093\/48990620\/btaa046.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/10\/3093\/48990620\/btaa046.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T07:36:20Z","timestamp":1722324980000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/36\/10\/3093\/5716331"}},"subtitle":[],"editor":[{"given":"Alfonso","family":"Valencia","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2020,1,27]]},"references-count":26,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2020,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btaa046","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2019.12.31.891895","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,5,15]]},"published":{"date-parts":[[2020,1,27]]}}}