{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T02:51:46Z","timestamp":1791859906342,"version":"4.3.4"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2023,3,9]],"date-time":"2023-03-09T00:00:00Z","timestamp":1678320000000},"content-version":"vor","delay-in-days":268,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","award":["ISF 1804\/16"],"award-info":[{"award-number":["ISF 1804\/16"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Cross-validation is the de facto standard for predictive model evaluation and selection. In proper use, it provides an unbiased estimate of a model's predictive performance. However, data sets often undergo various forms of data-dependent preprocessing, such as mean-centring, rescaling, dimensionality reduction and outlier removal. It is often believed that such preprocessing stages, if done in an unsupervised manner (that does not incorporate the class labels or response values) are generally safe to do prior to cross-validation. In this paper, we study three commonly practised preprocessing procedures prior to a regression analysis: (i) variance-based feature selection; (ii) grouping of rare categorical features; and (iii) feature rescaling. We demonstrate that unsupervised preprocessing can, in fact, introduce a substantial bias into cross-validation estimates and potentially hurt model selection. This bias may be either positive or negative and its exact magnitude depends on all the parameters of the problem in an intricate manner. Further research is needed to understand the real-world impact of this bias across different application domains, particularly when dealing with small sample sizes and high-dimensional data.<\/jats:p>","DOI":"10.1111\/rssb.12537","type":"journal-article","created":{"date-parts":[[2022,6,14]],"date-time":"2022-06-14T05:01:15Z","timestamp":1655182875000},"page":"1474-1502","source":"Crossref","is-referenced-by-count":44,"title":["On the Cross-Validation Bias due to Unsupervised Preprocessing"],"prefix":"10.1093","volume":"84","author":[{"given":"Amit","family":"Moscovich","sequence":"first","affiliation":[{"name":"Department of Statistics and Operations Research , Tel Aviv , Israel"},{"name":"Tel Aviv University , Tel Aviv , Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saharon","family":"Rosset","sequence":"additional","affiliation":[{"name":"Department of Statistics and Operations Research , Tel Aviv , Israel"},{"name":"Tel Aviv University , Tel Aviv , Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,6,14]]},"reference":[{"issue":"6385","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1126\/science.aar5169","article-title":"Predicting reaction performance in C\u2013N cross-coupling using machine learning","volume":"360","author":"Ahneman","year":"2018","journal-title":"Science"},{"issue":"10","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"6562","DOI":"10.1073\/pnas.102102699","article-title":"Selection bias in gene extraction on the basis of microarray gene-expression data","volume":"99","author":"Ambroise","year":"2002","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1214\/09-SS054","article-title":"A survey of cross-validation procedures for model selection","volume":"4","author":"Arlot","year":"2010","journal-title":"Statistics Surveys"},{"key":"2023030906300664100_","first-page":"499","article-title":"Stability and generalization","volume":"2","author":"Bousquet","year":"2002","journal-title":"Journal of Machine Learning Research"},{"key":"2023030906300664100_","volume-title":"Sharper bounds for uniformly stable algorithms","author":"Bousquet","year":"2020"},{"issue":"3","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1961189.1961199","article-title":"LIBSVM: a library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"issue":"6378","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"926","DOI":"10.1126\/science.aar3247","article-title":"Detection and localization of surgically resectable cancers with a multi-analyte blood test","volume":"359","author":"Cohen","year":"2018","journal-title":"Science"},{"issue":"6376","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1126\/science.aao7104","article-title":"Morphology, muscle capacity, skill, and maneuvering ability in hummingbirds","volume":"359","author":"Dakin","year":"2018","journal-title":"Science"},{"issue":"6322","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"eaaf8399","DOI":"10.1126\/science.aaf8399","article-title":"Tumor aneuploidy correlates with markers of immune evasion and with reduced response to immunotherapy","volume":"355","author":"Davoli","year":"2017","journal-title":"Science"},{"key":"2023030906300664100_","volume-title":"UCI Machine Learning Repository","author":"Dua","year":"2017"},{"issue":"1","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1023\/B:MACH.0000019805.88351.60","article-title":"Leave one out error, stability, and generalization of voting combinations of classifiers","volume":"55","author":"Evgeniou","year":"2004","journal-title":"Machine Learning"},{"key":"2023030906300664100_","volume-title":"Advances in Neural Information Processing Systems","author":"Feldman","year":"2018"},{"key":"2023030906300664100_","first-page":"1270","article-title":"Proceedings of Machine Learning Research","volume-title":"Proceedings of the Thirty-Second Conference on Learning Theory","author":"Feldman","year":"2019"},{"issue":"April","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.commatsci.2018.07.052","article-title":"A data-driven statistical model for predicting the critical temperature of a superconductor","volume":"154","author":"Hamidieh","year":"2018","journal-title":"Computational Materials Science"},{"key":"2023030906300664100_","first-page":"1225","article-title":"Proceedings of Machine Learning Research","volume-title":"Proceedings of The 33rd International Conference on Machine Learning","author":"Hardt","year":"2016"},{"key":"2023030906300664100_","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-19425-7","volume-title":"Regression modeling strategies","author":"Harrell","year":"2015"},{"key":"2023030906300664100_","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning","author":"Hastie","year":"2009"},{"issue":"1","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1186\/s12874-015-0088-9","article-title":"A measure of the impact of CV incompleteness on prediction error estimation with application to PCA and normalization","volume":"15","author":"Hornung","year":"2015","journal-title":"BMC Medical Research Methodology"},{"key":"2023030906300664100_","author":"Hsu","year":"2010"},{"key":"2023030906300664100_","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-7138-7","volume-title":"An introduction to statistical learning: with applications in R","author":"James","year":"2013"},{"issue":"4","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2382577.2382579","article-title":"Leakage in data mining: formulation, detection, and avoidance","volume":"6","author":"Kaufman","year":"2012","journal-title":"ACM Transactions on Knowledge Discovery from Data"},{"issue":"5","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v028.i05","article-title":"Building predictive models in R using the caret package","volume":"28","author":"Kuhn","year":"2008","journal-title":"Journal of Statistical Software"},{"key":"2023030906300664100_","volume-title":"Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles","author":"Kuhn","year":"2020"},{"issue":"6320","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1126\/science.aah7111","article-title":"CRISPRi-based genome-scale identification of functional long noncoding RNA loci in human cells","volume":"355","author":"Liu","year":"2017","journal-title":"Science"},{"issue":"34","key":"2023030906300664100_","first-page":"1","article-title":"MLlib: machine learning in apache spark","volume":"17","author":"Meng","year":"2016","journal-title":"Journal of Machine Learning Research"},{"issue":"6374","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1126\/science.aao0284","article-title":"Learning and attention reveal a general relationship between population activity and behavior","volume":"359","author":"Ni","year":"2018","journal-title":"Science"},{"key":"2023030906300664100_","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"302","DOI":"10.1109\/TIT.2019.2945779","article-title":"How much does your data exploration overfit? Controlling bias via information usage","volume":"66","author":"Russo","year":"2020","journal-title":"IEEE Transactions on Information Theory"},{"issue":"6392","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1024","DOI":"10.1126\/science.aar6851","article-title":"Ancient human parallel lineages within North America contributed to a coastal expansion","volume":"360","author":"Scheib","year":"2018","journal-title":"Science"},{"key":"2023030906300664100_","first-page":"2635","article-title":"Learnability, stability and uniform convergence","volume":"11","author":"Shalev-Shwartz","year":"2010","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1093\/jnci\/95.1.14","article-title":"Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification","volume":"95","author":"Simon","year":"2003","journal-title":"JNCI Journal of the National Cancer Institute"},{"issue":"9","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1550","DOI":"10.1101\/gr.169375.113","article-title":"MultiBLUP: improved SNP-based prediction for complex traits","volume":"24","author":"Speed","year":"2014","journal-title":"Genome Research"},{"issue":"1","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression the shrinkage and selection via Lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"Journal of the Royal Statistical Society. Series B: Statistical Methodology"},{"issue":"3","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1088\/1741-2560\/12\/3\/031001","article-title":"EEG artifact removal\u2014state-of-the-art and guidelines","volume":"12","author":"Urig\u00fcen","year":"2015","journal-title":"Journal of Neural Engineering"},{"key":"2023030906300664100_","volume-title":"R for data science","author":"Wickham","year":"2017"},{"issue":"3","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1007\/s10994-011-5268-1","article-title":"Robustness and generalization","volume":"86","author":"Xu","year":"2012","journal-title":"Machine Learning"},{"issue":"7","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"3561","DOI":"10.1109\/TIT.2010.2048503","article-title":"Robust regression and Lasso","volume":"56","author":"Xu","year":"2010","journal-title":"IEEE Transactions on Information Theory"},{"issue":"7","key":"2023030906300664100_","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1038\/ng.608","article-title":"Common SNPs explain a large proportion of the heritability for human height","volume":"42","author":"Yang","year":"2010","journal-title":"Nature Genetics"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/rssb.12537","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/rssb.12537","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/4\/1474\/49462949\/jrsssb_84_4_1474.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/4\/1474\/49462949\/jrsssb_84_4_1474.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,26]],"date-time":"2024-09-26T18:20:21Z","timestamp":1727374821000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/84\/4\/1474\/7073256"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,14]]},"references-count":38,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,6,14]]},"published-print":{"date-parts":[[2022,9,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12537","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,9]]},"published":{"date-parts":[[2022,6,14]]}}}