{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T21:12:06Z","timestamp":1788297126513,"version":"build-2803163510"},"reference-count":46,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2018,8,28]],"date-time":"2018-08-28T00:00:00Z","timestamp":1535414400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004360","name":"Swedish University of Agricultural Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004360","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010390","name":"SLU","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100010390","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002835","name":"Chalmers University of Technology","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002835","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001862","name":"Swedish Research Council Formas","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001862","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"publisher","award":["201306300047"],"award-info":[{"award-number":["201306300047"]}],"id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,3,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Validation of variable selection and predictive performance is crucial in construction of robust multivariate models that generalize well, minimize overfitting and facilitate interpretation of results. Inappropriate variable selection leads instead to selection bias, thereby increasing the risk of model overfitting and false positive discoveries. Although several algorithms exist to identify a minimal set of most informative variables (i.e. the minimal-optimal problem), few can select all variables related to the research question (i.e. the all-relevant problem). Robust algorithms combining identification of both minimal-optimal and all-relevant variables with proper cross-validation are urgently needed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We developed the MUVR algorithm to improve predictive performance and minimize overfitting and false positives in multivariate analysis. In the MUVR algorithm, minimal variable selection is achieved by performing recursive variable elimination in a repeated double cross-validation (rdCV) procedure. The algorithm supports partial least squares and random forest modelling, and simultaneously identifies minimal-optimal and all-relevant variable sets for regression, classification and multilevel analyses. Using three authentic omics datasets, MUVR yielded parsimonious models with minimal overfitting and improved model performance compared with state-of-the-art rdCV. Moreover, MUVR showed advantages over other variable selection algorithms, i.e. Boruta and VSURF, including simultaneous variable selection and validation scheme and wider applicability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Algorithms, data, scripts and tutorial are open source and available as an R package (\u2018MUVR\u2019) at https:\/\/gitlab.com\/CarlBrunius\/MUVR.git.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/bty710","type":"journal-article","created":{"date-parts":[[2018,8,24]],"date-time":"2018-08-24T07:09:52Z","timestamp":1535094592000},"page":"972-980","source":"Crossref","is-referenced-by-count":193,"title":["Variable selection and validation in multivariate modelling"],"prefix":"10.1093","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9709-3394","authenticated-orcid":false,"given":"Lin","family":"Shi","sequence":"first","affiliation":[{"name":"Department of Molecular Sciences, Swedish University of Agricultural Sciences, Uppsala SE-750 07, Sweden"},{"name":"Department of Biology and Biological Engineering, Food and Nutrition Science, Chalmers University of Technology, Gothenburg SE-412 96, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johan A","family":"Westerhuis","sequence":"additional","affiliation":[{"name":"Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam XH, The Netherlands"},{"name":"Metabolomics Center, North-West University, X6001, Potchefstroom, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johan","family":"Ros\u00e9n","sequence":"additional","affiliation":[{"name":"Swedish National Food Agency, Uppsala, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rikard","family":"Landberg","sequence":"additional","affiliation":[{"name":"Department of Molecular Sciences, Swedish University of Agricultural Sciences, Uppsala SE-750 07, Sweden"},{"name":"Department of Biology and Biological Engineering, Food and Nutrition Science, Chalmers University of Technology, Gothenburg SE-412 96, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3957-870X","authenticated-orcid":false,"given":"Carl","family":"Brunius","sequence":"additional","affiliation":[{"name":"Department of Biology and Biological Engineering, Food and Nutrition Science, Chalmers University of Technology, Gothenburg SE-412 96, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,8,28]]},"reference":[{"key":"2023013107262314400_bty710-B1","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1002\/cem.2793","article-title":"Unsupervised random forest: a tutorial with case studies","volume":"30","author":"Afanador","year":"2016","journal-title":"J. Chemom"},{"key":"2023013107262314400_bty710-B2","doi-asserted-by":"crossref","first-page":"6562","DOI":"10.1073\/pnas.102102699","article-title":"Selection bias in gene extraction on the basis of microarray gene-expression data","volume":"99","author":"Ambroise","year":"2002","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023013107262314400_bty710-B3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13321-014-0047-1","article-title":"Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation","volume":"6","author":"Baumann","year":"2014","journal-title":"J. Cheminform"},{"key":"2023013107262314400_bty710-B4","doi-asserted-by":"crossref","first-page":"1702","DOI":"10.1093\/bioinformatics\/btm162","article-title":"WilcoxCV: an R package for fast variable selection in cross-validation","volume":"23","author":"Boulesteix","year":"2007","journal-title":"Bioinformatics"},{"key":"2023013107262314400_bty710-B5","doi-asserted-by":"crossref","first-page":"22806","DOI":"10.1038\/srep22806","article-title":"Bacterial associations reveal spatial population dynamics in Anopheles gambiae mosquitoes","volume":"6","author":"Buck","year":"2016","journal-title":"Sci. Rep"},{"key":"2023013107262314400_bty710-B6","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1093\/bib\/bbq073","article-title":"An empirical assessment of validation practices for molecular classifiers","volume":"12","author":"Castaldi","year":"2011","journal-title":"Brief. Bioinform"},{"key":"2023013107262314400_bty710-B7","first-page":"2079","article-title":"On over-fitting in model selection and subsequent selection bias in performance evaluation","volume":"11","author":"Cawley","year":"2010","journal-title":"J. Mach. Learn. Res"},{"key":"2023013107262314400_bty710-B8","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1074\/mcp.M112.022566","article-title":"A critical assessment of feature selection methods for biomarker discovery in clinical proteomics","volume":"12","author":"Christin","year":"2013","journal-title":"Mol. Cell. Proteomics"},{"key":"2023013107262314400_bty710-B9","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1186\/1471-2105-12-33","article-title":"A genetic algorithm-Bayesian network approach for the analysis of metabolomics and spectroscopic data: application to the rapid detection of Bacillus spores and identification of Bacillus species","volume":"12","author":"Correa","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2023013107262314400_bty710-B10","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1002\/cem.1225","article-title":"Repeated double cross validation","volume":"23","author":"Filzmoser","year":"2009","journal-title":"J. Chemom"},{"key":"2023013107262314400_bty710-B11","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.micres.2015.01.003","article-title":"Multi -omics and metabolic modelling pipelines: challenges and tools for systems microbiology","volume":"171","author":"Fondi","year":"2015","journal-title":"Microbiol. Res"},{"key":"2023013107262314400_bty710-B12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10661-017-6025-0","article-title":"Assessing the accuracy and stability of variable selection methods for random forest modeling in ecology","volume":"189","author":"Fox","year":"2017","journal-title":"Environ. Monit. Assess"},{"key":"2023013107262314400_bty710-B13","volume-title":"R J. Journal","author":"Genuer","year":"2015"},{"key":"2023013107262314400_bty710-B14","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1016\/j.trac.2016.07.004","article-title":"Data analysis strategies for targeted and untargeted LC-MS metabolomic studies: overview and workflow","volume":"82","author":"Gorrochategui","year":"2016","journal-title":"TrAC Trends Anal. Chem"},{"key":"2023013107262314400_bty710-B15","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.csda.2015.04.002","article-title":"Grouped variable importance with random forests and application to multiple functional data analysis","volume":"90","author":"Gregorutti","year":"2015","journal-title":"Comput. Stat. Data Anal"},{"key":"2023013107262314400_bty710-B16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.aca.2014.03.039","article-title":"A comparative investigation of modern feature selection and classification approaches for the analysis of mass spectrometry data","volume":"829","author":"Gromski","year":"2014","journal-title":"Anal. Chim. Acta"},{"key":"2023013107262314400_bty710-B17","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.aca.2015.02.012","article-title":"A tutorial review: metabolomics and partial least squares-discriminant analysis \u2013 a marriage of convenience or a shotgun wedding","volume":"879","author":"Gromski","year":"2015","journal-title":"Anal. Chim. Acta"},{"key":"2023013107262314400_bty710-B18","doi-asserted-by":"crossref","first-page":"2315","DOI":"10.1002\/mnfr.201500423","article-title":"Discovery of urinary biomarkers of whole grain rye intake in free-living subjects using nontargeted LC-MS metabolite profiling","volume":"59","author":"Hanhineva","year":"2015","journal-title":"Mol. Nutr. Food Res"},{"key":"2023013107262314400_bty710-B19","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.csda.2012.09.020","article-title":"A new variable selection approach using Random Forests","volume":"60","author":"Hapfelmeier","year":"2013","journal-title":"Comput. Stat. Data Anal"},{"key":"2023013107262314400_bty710-B20","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.artmed.2015.11.001","article-title":"The feature selection bias problem in relation to high-dimensional gene data","volume":"66","author":"Krawczuk","year":"2016","journal-title":"Artif. Intell. Med"},{"key":"2023013107262314400_bty710-B21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1758-2946-6-10","article-title":"Cross-validation pitfalls when selecting and assessing regression and classification models","volume":"6","author":"Krstajic","year":"2014","journal-title":"J. Cheminform"},{"key":"2023013107262314400_bty710-B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v036.i11","article-title":"Feature selection with the Boruta Package","volume":"36","author":"Kursa","year":"2010","journal-title":"J. Stat. Softw"},{"key":"2023013107262314400_bty710-B23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s12064-012-0168-x","article-title":"Systems genetics in \u2018-omics\u2019 era: current and future development","volume":"132","author":"Li","year":"2013","journal-title":"Theory Biosci"},{"key":"2023013107262314400_bty710-B24","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1002\/(SICI)1099-128X(199609)10:5\/6<521::AID-CEM448>3.0.CO;2-J","article-title":"Model validation by permutation tests","volume":"10","author":"Lindgren","year":"1996","journal-title":"J. Chemom"},{"key":"2023013107262314400_bty710-B25","doi-asserted-by":"crossref","DOI":"10.1186\/1748-7188-6-27","article-title":"A Partial Least Squares based algorithm for parsimonious variable selection","volume":"6","author":"Mehmood","year":"2011","journal-title":"Algorithms Mol. Biol"},{"key":"2023013107262314400_bty710-B26","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.chemolab.2012.07.010","article-title":"A review of variable selection methods in Partial Least Squares Regression","volume":"118","author":"Mehmood","year":"2012","journal-title":"Chemom. Intell. Lab. Syst"},{"key":"2023013107262314400_bty710-B27","first-page":"628","volume-title":"Brief Bioinform","author":"Meng","year":"2016"},{"key":"2023013107262314400_bty710-B28","first-page":"589","article-title":"Consistent feature selection for pattern recognition in polynomial time","volume":"8","author":"Nilsson","year":"2007","journal-title":"J. Mach. Learn. Res"},{"key":"2023013107262314400_bty710-B29","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1038\/nrm3314","article-title":"Metabolomics: the apogee of the omics trilogy","volume":"13","author":"Patti","year":"2012","journal-title":"Nat. Rev. Mol. Cell Biol"},{"key":"2023013107262314400_bty710-B30","first-page":"596","article-title":"On the dangers of cross-validation an experimental evaluation","volume":"588","author":"Rao","year":"2006","journal-title":"Solutions"},{"key":"2023013107262314400_bty710-B31","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-662-45620-0_2","article-title":"All Relevant Feature Selection Methods and Applications","volume-title":"Feature Selection for Data and Pattern Recognition. Studies in Computational Intelligence","author":"Rudnicki","year":"2015"},{"key":"2023013107262314400_bty710-B32","doi-asserted-by":"crossref","first-page":"2507","DOI":"10.1093\/bioinformatics\/btm344","article-title":"A review of feature selection techniques in bioinformatics","volume":"23","author":"Saeys","year":"2007","journal-title":"Bioinformatics"},{"key":"2023013107262314400_bty710-B33","first-page":"1","article-title":"Robustness of Random Forest-based gene selection methods","volume":"23","author":"Saeys","year":"2014","journal-title":"Bioinformatics"},{"key":"2023013107262314400_bty710-B34","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1007\/s00125-017-4521-y","article-title":"Plasma metabolites associated with type 2 diabetes in a Swedish population: a case\u2013control study nested in a prospective cohort","volume":"61","author":"Shi","year":"2018","journal-title":"Diabetologia"},{"key":"2023013107262314400_bty710-B35","doi-asserted-by":"crossref","first-page":"1600924","DOI":"10.1002\/mnfr.201600924","article-title":"Targeted metabolomics reveals differences in the extended postprandial plasma metabolome of healthy subjects after intake of whole-grain rye porridges versus refined wheat bread","volume":"61","author":"Shi","year":"2017","journal-title":"Mol. Nutr. Food Res"},{"key":"2023013107262314400_bty710-B36","doi-asserted-by":"crossref","first-page":"S9.","DOI":"10.1186\/1471-2105-15-S7-S9","article-title":"Proteomics, lipidomics, metabolomics: a mass spectrometry tutorial from a computer scientist\u2019s point of view","volume":"15","author":"Smith","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2023013107262314400_bty710-B37","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1186\/1471-2105-8-25","article-title":"Bias in random forest variable importance measures: illustrations, sources and a solution","volume":"8","author":"Strobl","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023013107262314400_bty710-B38","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1007\/978-1-61779-027-0_23","article-title":"Omics-based identification of pathophysiological processes","volume":"719","author":"Tanaka","year":"2011","journal-title":"Methods Mol. Biol"},{"key":"2023013107262314400_bty710-B39","article-title":"Model Comparison and the Principle of Parsimony","author":"Vandekerckhove","year":"2014","journal-title":"Oxford Handbook of Computational and Mathematical Psychology"},{"key":"2023013107262314400_bty710-B40","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1016\/j.neuroimage.2016.10.038","article-title":"Assessing and tuning brain decoders: cross-validation, caveats, and guidelines","volume":"145","author":"Varoquaux","year":"2017","journal-title":"Neuroimage"},{"key":"2023013107262314400_bty710-B41","article-title":"Cross-validation failure: small sample sizes lead to large error bars","author":"Varoquaux","year":"2017"},{"key":"2023013107262314400_bty710-B42","doi-asserted-by":"crossref","first-page":"4483","DOI":"10.1021\/pr800145j","article-title":"Multilevel Data Analysis of a Crossover Designed Human Nutritional Intervention Study research articles","volume":"7","author":"Van Velzen","year":"2008","journal-title":"J. Proteome Res"},{"key":"2023013107262314400_bty710-B43","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s11306-009-0185-z","article-title":"Multivariate paired data analysis: multilevel PLSDA versus OPLSDA","volume":"6","author":"Westerhuis","year":"2010","journal-title":"Metabolomics"},{"key":"2023013107262314400_bty710-B44","first-page":"1","volume-title":"J. Stat. Softw","author":"Wright","year":"2015"},{"key":"2023013107262314400_bty710-B45","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.aca.2016.02.001","article-title":"Chemometric methods in data processing of mass spectrometry-based metabolomics: a review","volume":"914","author":"Yi","year":"2016","journal-title":"Anal. Chim. Acta"},{"key":"2023013107262314400_bty710-B46","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1017\/S000711451700263X","article-title":"Impact of sourdough fermentation on appetite and postprandial metabolic responses \u2013 a randomised cross-over trial with whole grain rye crispbread","volume":"118","author":"Zamaratskaia","year":"2017","journal-title":"Br. J. Nutr"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/6\/972\/48967176\/bioinformatics_35_6_972.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/6\/972\/48967176\/bioinformatics_35_6_972.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T05:24:24Z","timestamp":1675142664000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/35\/6\/972\/5085367"}},"subtitle":[],"editor":[{"given":"Janet","family":"Kelso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2018,8,28]]},"references-count":46,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2019,3,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bty710","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,3,15]]},"published":{"date-parts":[[2018,8,28]]}}}