{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T13:21:32Z","timestamp":1790774492114,"version":"4.1.0"},"reference-count":84,"publisher":"Springer Science and Business Media LLC","issue":"8","license":[{"start":{"date-parts":[[2021,8,1]],"date-time":"2021-08-01T00:00:00Z","timestamp":1627776000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T00:00:00Z","timestamp":1627862400000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N62909-19-1-2096"],"award-info":[{"award-number":["N62909-19-1-2096"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["437611051"],"award-info":[{"award-number":["437611051"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    We propose the conditional predictive impact (CPI), a consistent and unbiased estimator of the association between one or several features and a given outcome, conditional on a reduced feature set. Building on the knockoff framework of Cand\u00e8s et al. (J R Stat Soc Ser B 80:551\u2013577, 2018), we develop a novel testing procedure that works in conjunction with any valid knockoff sampler, supervised learning algorithm, and loss function. The CPI can be efficiently computed for high-dimensional data without any sparsity constraints. We demonstrate convergence criteria for the CPI and develop statistical inference procedures for evaluating its magnitude, significance, and precision. These tests aid in feature and model selection, extending traditional frequentist and Bayesian techniques to general supervised learning tasks. The CPI may also be applied in causal discovery to identify underlying multivariate graph structures. We test our method using various algorithms, including linear regression, neural networks, random forests, and support vector machines. Empirical results show that the CPI compares favorably to alternative variable importance measures and other nonparametric tests of conditional independence on a diverse array of real and synthetic datasets. Simulations confirm that our inference procedures successfully control Type I error with competitive power in a range of settings. Our method has been implemented in an package, , which can be downloaded from\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/dswatson\/cpi\">https:\/\/github.com\/dswatson\/cpi<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1007\/s10994-021-06030-6","type":"journal-article","created":{"date-parts":[[2021,8,2]],"date-time":"2021-08-02T17:03:36Z","timestamp":1627923816000},"page":"2107-2129","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":58,"title":["Testing conditional independence in supervised learning algorithms"],"prefix":"10.1007","volume":"110","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9632-2159","authenticated-orcid":false,"given":"David S.","family":"Watson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marvin N.","family":"Wright","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,8,2]]},"reference":[{"issue":"7","key":"6030_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 1\u201346.","journal-title":"PLoS ONE"},{"issue":"5","key":"6030_CR2","doi-asserted-by":"publisher","first-page":"2055","DOI":"10.1214\/15-AOS1337","volume":"43","author":"RF Barber","year":"2015","unstructured":"Barber, R. F., & Cand\u00e8s, E. J. (2015). Controlling the false discovery rate via knockoffs. Annals of Statistics, 43(5), 2055\u20132085.","journal-title":"Annals of Statistics"},{"key":"6030_CR3","doi-asserted-by":"crossref","unstructured":"Bates, S., Cand\u00e8s, E., Janson, L., & Wang, W. (2020). Metropolized knockoff sampling. Journal of the American Statistical Association, 1\u201315.","DOI":"10.1080\/01621459.2020.1729163"},{"issue":"1","key":"6030_CR4","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","volume":"57","author":"Y Benjamini","year":"1995","unstructured":"Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (statistical Methodology), 57(1), 289\u2013300.","journal-title":"Journal of the Royal Statistical Society: Series B (statistical Methodology)"},{"issue":"4","key":"6030_CR5","doi-asserted-by":"publisher","first-page":"1165","DOI":"10.1214\/aos\/1013699998","volume":"29","author":"Y Benjamini","year":"2001","unstructured":"Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165\u20131188.","journal-title":"Annals of Statistics"},{"issue":"1","key":"6030_CR6","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1111\/rssb.12340","volume":"82","author":"TB Berrett","year":"2020","unstructured":"Berrett, T. B., Wang, Y., Barber, R. F., & Samworth, R. J. (2020). The conditional permutation test for independence while controlling for confounders. Journal of the Royal Statistical Society: Series B (statistical Methodology), 82(1), 175\u2013197. https:\/\/doi.org\/10.1111\/rssb.12340","journal-title":"Journal of the Royal Statistical Society: Series B (statistical Methodology)"},{"issue":"170","key":"6030_CR7","first-page":"1","volume":"17","author":"B Bischl","year":"2016","unstructured":"Bischl, B., Lang, M., Kotthoff, L., Schiffner, J., Richter, J., Studerus, E., et al. (2016). mlr: Machine learning in R. Journal of Machine Learning Research, 17(170), 1\u20135.","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"6030_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman, L. (2001). Random forests. Machine Learning, 45(1), 1\u201333.","journal-title":"Machine Learning"},{"issue":"3","key":"6030_CR9","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1111\/rssb.12265","volume":"80","author":"E Cand\u00e8s","year":"2018","unstructured":"Cand\u00e8s, E., Fan, Y., Janson, L., & Lv, J. (2018). Panning for gold: \u2018model-X\u2019 knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (statistical Methodology), 80(3), 551\u2013577.","journal-title":"Journal of the Royal Statistical Society: Series B (statistical Methodology)"},{"key":"6030_CR11","unstructured":"Doran, G., Muandet, K., Zhang, K., & Sch\u00f6lkopf, B. (2014). A permutation-based kernel conditional independence test. In Proceedings of the International Conference on Uncertainty in Artificial Intelligence (pp. 132\u2013141)."},{"key":"6030_CR12","volume-title":"UCI machine learning repository","author":"D Dua","year":"2017","unstructured":"Dua, D., & Graff, C. (2017). UCI machine learning repository. University of California, School of Information and Computer Science."},{"key":"6030_CR13","unstructured":"Feng, J., Williamson, B., Simon, N., & Carone, M. (2018). Nonparametric variable importance using an augmented neural network with multi-task learning. In Proceedings of the International Conference on Machine Learning (pp. 1496\u20131505)."},{"key":"6030_CR14","volume-title":"The design of experiments","author":"RA Fisher","year":"1935","unstructured":"Fisher, R. A. (1935). The design of experiments. Oliver & Boyd."},{"issue":"177","key":"6030_CR15","first-page":"1","volume":"20","author":"A Fisher","year":"2019","unstructured":"Fisher, A., Rudin, C., & Dominici, F. (2019). All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20(177), 1\u201381.","journal-title":"Journal of Machine Learning Research"},{"key":"6030_CR16","first-page":"1531","volume":"5","author":"F Fleuret","year":"2004","unstructured":"Fleuret, F. (2004). Fast binary feature selection with conditional mutual information. Journal of Machine Learning Research, 5, 1531\u20131555.","journal-title":"Journal of Machine Learning Research"},{"issue":"3","key":"6030_CR17","doi-asserted-by":"publisher","first-page":"916","DOI":"10.1214\/07-AOAS148","volume":"2","author":"JH Friedman","year":"2008","unstructured":"Friedman, J. H., & Popescu, B. E. (2008). Predictive learning via rule ensembles. The Annals of Applied Statistics, 2(3), 916\u2013954.","journal-title":"The Annals of Applied Statistics"},{"issue":"1","key":"6030_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v033.i01","volume":"33","author":"J Friedman","year":"2010","unstructured":"Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1\u201341.","journal-title":"Journal of Statistical Software"},{"key":"6030_CR19","first-page":"489","volume":"20","author":"K Fukumizu","year":"2008","unstructured":"Fukumizu, K., Gretton, A., Sun, X., & Sch\u00f6lkopf, B. (2008). Kernel measures of conditional dependence. Advances in Neural Information Processing Systems, 20, 489\u2013496.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"3","key":"6030_CR20","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1016\/S0304-3800(02)00257-0","volume":"160","author":"M Gevrey","year":"2003","unstructured":"Gevrey, M., Dimopoulos, I., & Lek, S. (2003). Review and comparison of methods to study the contribution of variables in artificial neural network models. Ecological Modelling, 160(3), 249\u2013264.","journal-title":"Ecological Modelling"},{"key":"6030_CR21","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.csda.2015.04.002","volume":"90","author":"B Gregorutti","year":"2015","unstructured":"Gregorutti, B., Michel, B., & Saint-Pierre, P. (2015). Grouped variable importance with random forests and application to multiple functional data analysis. Computational Statistics & Data Analysis, 90, 15\u201335.","journal-title":"Computational Statistics & Data Analysis"},{"issue":"2","key":"6030_CR22","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1198\/000313007X188252","volume":"61","author":"U Gr\u00f6mping","year":"2007","unstructured":"Gr\u00f6mping, U. (2007). Estimators of relative importance in linear regression based on variance decomposition. The American Statistician, 61(2), 139\u2013147.","journal-title":"The American Statistician"},{"key":"6030_CR23","unstructured":"Guedj, B. (2019). A primer on PAC-Bayesian learning. arXiv preprint, 1901.05353."},{"issue":"7\/8","key":"6030_CR24","first-page":"1157","volume":"3","author":"I Guyon","year":"2003","unstructured":"Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3(7\/8), 1157\u20131182.","journal-title":"Journal of Machine Learning Research"},{"key":"6030_CR25","unstructured":"Hansen, D., Manzo, B., & Regier, J. (2021). Normalizing flows for knockoff-free controlled feature selection. arXiv preprint, 2106.01528."},{"issue":"1","key":"6030_CR26","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/0095-0696(78)90006-2","volume":"5","author":"D Harrison","year":"1978","unstructured":"Harrison, D., & Rubinfeld, D. L. (1978). Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1), 81\u2013102.","journal-title":"Journal of Environmental Economics and Management"},{"issue":"5","key":"6030_CR27","doi-asserted-by":"publisher","first-page":"R76","DOI":"10.1186\/gb-2007-8-5-r76","volume":"8","author":"JI Herschkowitz","year":"2007","unstructured":"Herschkowitz, J. I., Simin, K., Weigman, V. J., Mikaelian, I., Usary, J., Hu, Z., et al. (2007). Identification of conserved gene expression features between murine mammary carcinoma models and human breast tumors. Genome Biology, 8(5), R76.","journal-title":"Genome Biology"},{"issue":"2","key":"6030_CR28","first-page":"65","volume":"6","author":"S Holm","year":"1979","unstructured":"Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65\u201370.","journal-title":"Scandinavian Journal of Statistics"},{"key":"6030_CR29","first-page":"125","volume-title":"Targeted learning in data science","author":"AE Hubbard","year":"2018","unstructured":"Hubbard, A. E., Kennedy, C. J., & van der Laan, M. J. (2018). Data-adaptive target parameters. In M. J. van der Laan & S. Rose (Eds.), Targeted learning in data science (pp. 125\u2013142). Springer."},{"issue":"11","key":"6030_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v047.i11","volume":"47","author":"M Kalisch","year":"2012","unstructured":"Kalisch, M., M\u00e4chler, M., Colombo, D., Maathuis, M. H., & B\u00fchlmann, P. (2012). Causal inference using graphical models with the R package pcalg. Journal of Statistical Software, 47(11), 1\u201326.","journal-title":"Journal of Statistical Software"},{"key":"6030_CR31","volume-title":"Probabilistic graphical models: Principles and techniques","author":"D Koller","year":"2009","unstructured":"Koller, D., & Friedman, N. (2009). Probabilistic graphical models: Principles and techniques. MIT Press."},{"key":"6030_CR32","volume-title":"Bayesian artificial Intelligence","author":"KB Korb","year":"2009","unstructured":"Korb, K. B., & Nicholson, A. E. (2009). Bayesian artificial Intelligence (2nd ed.). Chapman and Hall\/CRC.","edition":"2"},{"issue":"2","key":"6030_CR88","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1037\/a0029146","volume":"142","author":"JK Kruschke","year":"2013","unstructured":"Kruschke J. K. (2013). Bayesian estimation supersedes the t test. Journal of Experimental Psychology: General, 142(2), 573\u2013603. https:\/\/doi.org\/10.1037\/a0029146","journal-title":"Journal of Experimental Psychology: General"},{"key":"6030_CR33","doi-asserted-by":"publisher","DOI":"10.1201\/9781315108230","volume-title":"Feature engineering and selection: A practical approach for predictive models","author":"M Kuhn","year":"2019","unstructured":"Kuhn, M., & Johnson, K. (2019). Feature engineering and selection: A practical approach for predictive models. Chapman and Hall\/CRC."},{"issue":"11","key":"6030_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v036.i11","volume":"36","author":"MB Kursa","year":"2010","unstructured":"Kursa, M. B., & Rudnicki, W. R. (2010). Feature selection with the Boruta package. Journal of Statistical Software, 36(11), 1\u201313.","journal-title":"Journal of Statistical Software"},{"issue":"523","key":"6030_CR35","doi-asserted-by":"publisher","first-page":"1094","DOI":"10.1080\/01621459.2017.1307116","volume":"113","author":"J Lei","year":"2018","unstructured":"Lei, J., G\u2019Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523), 1094\u20131111.","journal-title":"Journal of the American Statistical Association"},{"key":"6030_CR36","doi-asserted-by":"publisher","first-page":"907","DOI":"10.1038\/nm.2000","volume":"15","author":"E Lim","year":"2009","unstructured":"Lim, E., Vaillant, F., Wu, D., Forrest, N. C., Pal, B., Hart, A. H., et al. (2009). Aberrant luminal progenitors as the candidate target population for basal tumor development in BRCA1 mutation carriers. Nature Medicine, 15, 907.","journal-title":"Nature Medicine"},{"key":"6030_CR37","volume-title":"Introduction to bivariate and multivariate analysis","author":"RH Lindeman","year":"1980","unstructured":"Lindeman, R. H., Merenda, P. F., & Gold, R. Z. (1980). Introduction to bivariate and multivariate analysis. Longman."},{"key":"6030_CR38","first-page":"4765","volume":"30","author":"SM Lundberg","year":"2017","unstructured":"Lundberg, S. M., & Lee, S. -I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 4765\u20134774.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"6A","key":"6030_CR39","doi-asserted-by":"publisher","first-page":"3133","DOI":"10.1214\/09-AOS685","volume":"37","author":"MH Maathuis","year":"2009","unstructured":"Maathuis, M. H., Kalisch, M., & B\u00fchlmann, P. (2009). Estimating high-dimensional intervention effects from observational data. Annals of Statistics, 37(6A), 3133\u20133164.","journal-title":"Annals of Statistics"},{"issue":"6","key":"6030_CR40","doi-asserted-by":"publisher","first-page":"2068","DOI":"10.1016\/j.patcog.2009.12.013","volume":"43","author":"J Mart\u00ednez Sotoca","year":"2010","unstructured":"Mart\u00ednez Sotoca, J., & Pla, F. (2010). Supervised feature selection by clustering using conditional mutual information-based distances. Pattern Recognition, 43(6), 2068\u20132081.","journal-title":"Pattern Recognition"},{"issue":"4","key":"6030_CR41","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1111\/j.1467-9868.2010.00740.x","volume":"72","author":"N Meinshausen","year":"2010","unstructured":"Meinshausen, N., & B\u00fchlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society: Series B (statistical Methodology), 72(4), 417\u2013473.","journal-title":"Journal of the Royal Statistical Society: Series B (statistical Methodology)"},{"issue":"1","key":"6030_CR42","first-page":"841","volume":"17","author":"L Mentch","year":"2016","unstructured":"Mentch, L., & Hooker, G. (2016). Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. Journal of Machine Learning Research, 17(1), 841\u2013881.","journal-title":"Journal of Machine Learning Research"},{"key":"6030_CR43","unstructured":"Meyer, D., Dimitriadou, E., Hornik, K., Weingessel, A., & Leisch, F. (2018). e1071: Misc functions of the department of statistics, probability theory group. CRAN. R package version 1.7\u20130."},{"issue":"1","key":"6030_CR44","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1186\/1471-2105-11-110","volume":"11","author":"KK Nicodemus","year":"2010","unstructured":"Nicodemus, K. K., Malley, J. D., Strobl, C., & Ziegler, A. (2010). The behaviour of random forest permutation-based variable importance measures under predictor correlation. BMC Bioinformatics, 11(1), 110.","journal-title":"BMC Bioinformatics"},{"key":"6030_CR45","unstructured":"Patterson, E., & Sesia, M. (2018). knockoff. CRAN. R package version 0.3.2."},{"key":"6030_CR46","volume-title":"Probabilistic reasoning in intelligent systems: Networks of plausible inference","author":"J Pearl","year":"1988","unstructured":"Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann."},{"key":"6030_CR47","doi-asserted-by":"crossref","unstructured":"Phipson, B., & Smyth, G. (2010). Permutation P-values should never be zero: Calculating exact P-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(1).","DOI":"10.2202\/1544-6115.1585"},{"key":"6030_CR49","unstructured":"Ramsey, J. D. (2014). A scalable conditional independence test for nonlinear, non-Gaussian data. arXiv preprint, arXiv:1401.5031"},{"key":"6030_CR50","doi-asserted-by":"crossref","unstructured":"Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). \u201cWhy should I trust you?\u201d: Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135\u20131144).","DOI":"10.1145\/2939672.2939778"},{"issue":"6","key":"6030_CR51","doi-asserted-by":"publisher","first-page":"3438","DOI":"10.1214\/18-AOS1784","volume":"47","author":"A Rinaldo","year":"2019","unstructured":"Rinaldo, A., Wasserman, L., & G\u2019Sell, M. (2019). Bootstrapping and sample splitting for high-dimensional, assumption-lean inference. Annals of Statistics, 47(6), 3438\u20133469.","journal-title":"Annals of Statistics"},{"issue":"532","key":"6030_CR52","doi-asserted-by":"publisher","first-page":"1861","DOI":"10.1080\/01621459.2019.1660174","volume":"115","author":"Y Romano","year":"2020","unstructured":"Romano, Y., Sesia, M., Cand\u00e8s, E. (2020). Deep Knockoffs. Journal of the American Statistical Association, 115(532) 1861\u20131872. https:\/\/doi.org\/10.1080\/01621459.2019.1660174","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"6030_CR85","doi-asserted-by":"publisher","first-page":"225","DOI":"10.3758\/PBR.16.2.225","volume":"16","author":"JN Rouder","year":"2009","unstructured":"Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225\u2013237. https:\/\/doi.org\/10.3758\/PBR.16.2.225","journal-title":"Psychonomic Bulletin & Review"},{"issue":"1","key":"6030_CR53","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/0097-3165(72)90019-2","volume":"13","author":"N Sauer","year":"1972","unstructured":"Sauer, N. (1972). On the density of families of sets. Journal of Combinatorial Theory Series A, 13(1), 145\u2013147.","journal-title":"Journal of Combinatorial Theory Series A"},{"issue":"3","key":"6030_CR54","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v035.i03","volume":"35","author":"M Scutari","year":"2010","unstructured":"Scutari, M. (2010). Learning Bayesian networks with the bnlearnR package. Journal of Statistical Software, 35(3), 1\u201322.","journal-title":"Journal of Statistical Software"},{"key":"6030_CR55","doi-asserted-by":"publisher","DOI":"10.1201\/b17065","volume-title":"Bayesian networks: With examples in R","author":"M Scutari","year":"2014","unstructured":"Scutari, M., & Denis, J.-B. (2014). Bayesian networks: With examples in R. Chapman and Hall\/CRC."},{"issue":"1","key":"6030_CR56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/biomet\/asy033","volume":"106","author":"M Sesia","year":"2019","unstructured":"Sesia, M., Sabatti, C., & Cand\u00e8s, E. J. (2019). Gene hunting with hidden Markov model knockoffs. Biometrika, 106(1), 1\u201318.","journal-title":"Biometrika"},{"issue":"3","key":"6030_CR57","doi-asserted-by":"publisher","first-page":"1514","DOI":"10.1214\/19-AOS1857","volume":"48","author":"R Shah","year":"2020","unstructured":"Shah, R., & Peters, J. (2020). The hardness of conditional independence testing and the generalised covariance measure. Annals of Statistics, 48(3), 1514\u20131538.","journal-title":"Annals of Statistics"},{"key":"6030_CR58","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781107298019","volume-title":"Understanding machine learning: From theory to algorithms","author":"S Shalev-Shwartz","year":"2014","unstructured":"Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding machine learning: From theory to algorithms. Cambridge University Press."},{"issue":"1","key":"6030_CR59","doi-asserted-by":"publisher","first-page":"247","DOI":"10.2140\/pjm.1972.41.247","volume":"41","author":"S Shelah","year":"1972","unstructured":"Shelah, S. (1972). A combinatorial problem: Stability and orders for models and theories in infinitariy languages. Pacific Journal of Mathematics, 41(1), 247\u2013261.","journal-title":"Pacific Journal of Mathematics"},{"key":"6030_CR60","unstructured":"Shrikumar, A., Greenside, P., & Kundaje, A. (2017). Learning important features through propagating activation differences. In Proceedings of the International Conference on Machine Learning (Vol. 70, pp. 3145\u20133153)."},{"issue":"14","key":"6030_CR61","doi-asserted-by":"publisher","first-page":"8418","DOI":"10.1073\/pnas.0932692100","volume":"100","author":"T S\u00f8rlie","year":"2003","unstructured":"S\u00f8rlie, T., Tibshirani, R., Parker, J., Hastie, T., Marron, J. S., Nobel, A., et al. (2003). Repeated observation of breast tumor subtypes in independent gene expression data sets. Proceedings of the National Academy of Sciences, 100(14), 8418\u20138423.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"6030_CR62","volume-title":"Causation, prediction, and search","author":"P Spirtes","year":"2000","unstructured":"Spirtes, P., Glymour, C. N., & Scheines, R. (2000). Causation, prediction, and search (2nd ed.). The MIT Press.","edition":"2"},{"key":"6030_CR63","unstructured":"Steinke, T., & Zakynthinou, L. (2020). Reasoning about generalization via conditional mutual information. In Proceedings of the International Conference on Learning Theory (pp. 3437\u20133452)."},{"issue":"3","key":"6030_CR64","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1111\/1467-9868.00346","volume":"64","author":"JD Storey","year":"2002","unstructured":"Storey, J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B (statistical Methodology), 64(3), 479\u2013498.","journal-title":"Journal of the Royal Statistical Society: Series B (statistical Methodology)"},{"issue":"1","key":"6030_CR65","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1186\/1471-2105-9-307","volume":"9","author":"C Strobl","year":"2008","unstructured":"Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., & Zeileis, A. (2008). Conditional variable importance for random forests. BMC Bioinformatics, 9(1), 307.","journal-title":"BMC Bioinformatics"},{"key":"6030_CR66","doi-asserted-by":"crossref","unstructured":"Strobl, E. V., Zhang, K., & Visweswaran, S. (2018). Approximate kernel-based conditional independence tests for fast non-parametric causal discovery. Journal of Causal Inference, 7(1), 20180017.","DOI":"10.1515\/jci-2018-0017"},{"issue":"43","key":"6030_CR67","doi-asserted-by":"publisher","first-page":"15545","DOI":"10.1073\/pnas.0506580102","volume":"102","author":"A Subramanian","year":"2005","unstructured":"Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., & Gillette, M. A. (2005). Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences, 102(43), 15545\u201315550.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"6030_CR68","doi-asserted-by":"crossref","unstructured":"Tansey, W., Veitch, V., Zhang, H., Rabadan, R., & Blei, D.M. (2021). The holdout randomization test for feature selection in black box models. Journal of Computational and Graphical Statistics, 1\u201337.","DOI":"10.1080\/10618600.2021.1923520"},{"key":"6030_CR48","unstructured":"Team, R. C. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing."},{"issue":"1","key":"6030_CR69","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","volume":"58","author":"R Tibshirani","year":"1996","unstructured":"Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B, 58(1), 267\u2013288.","journal-title":"Journal of the Royal Statistical Society B"},{"key":"6030_CR70","doi-asserted-by":"publisher","first-page":"5846","DOI":"10.1038\/sj.onc.1209876","volume":"25","author":"NC Turner","year":"2006","unstructured":"Turner, N. C., & Reis-Filho, J. S. (2006). Basal-like breast cancer and the BRCA1 phenotype. Oncogene, 25, 5846.","journal-title":"Oncogene"},{"key":"6030_CR71","doi-asserted-by":"crossref","unstructured":"van der Laan, M. J. (2006). Statistical inference for variable importance. The International Journal of Biostatistics, 2(1).","DOI":"10.2202\/1557-4679.1008"},{"key":"6030_CR72","volume-title":"Targeted learning in data science: Causal inference for complex longitudinal studies","year":"2018","unstructured":"van der Laan, M. J., & Rose, S. (Eds.). (2018). Targeted learning in data science: Causal inference for complex longitudinal studies. Springer."},{"issue":"2","key":"6030_CR73","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1137\/1116025","volume":"16","author":"V Vapnik","year":"1971","unstructured":"Vapnik, V., & Chervonenkis, A. (1971). On the uniform convergence of relative frequencies to their probabilities. Theory of Probabability & Its Applications, 16(2), 264\u2013280.","journal-title":"Theory of Probabability & Its Applications"},{"issue":"2","key":"6030_CR74","doi-asserted-by":"publisher","first-page":"26214","DOI":"10.1103\/PhysRevE.77.026214","volume":"77","author":"M Vejmelka","year":"2008","unstructured":"Vejmelka, M., & Palu\u0161, M. (2008). Inferring the directionality of coupling with conditional mutual information. Physical Review E, 77(2), 26214.","journal-title":"Physical Review E"},{"key":"6030_CR75","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-21706-2","volume-title":"Modern applied statistics with S","author":"WN Venables","year":"2002","unstructured":"Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). Springer.","edition":"4"},{"key":"6030_CR76","unstructured":"Verma, T., & Pearl, J. (1991). Equivalence and synthesis of causal models. In Proceedings of the International Conference on Uncertainty in Artificial Intelligence (pp. 255\u2013270)."},{"issue":"2","key":"6030_CR77","first-page":"841","volume":"31","author":"S Wachter","year":"2018","unstructured":"Wachter, S., Mittelstadt, B., & Russell, C. (2018). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31(2), 841\u2013887.","journal-title":"Harvard Journal of Law & Technology"},{"issue":"4","key":"6030_CR87","doi-asserted-by":"publisher","first-page":"752","DOI":"10.3758\/PBR.16.4.752","volume":"16","author":"R Wetzels","year":"2009","unstructured":"Wetzels, R., Raaijmakers, J. G. W., Jakab, E., Wagenmakers, E.-J. (2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin & Review, 16(4), 752\u2013760. https:\/\/doi.org\/10.3758\/PBR.16.4.752","journal-title":"Psychonomic Bulletin & Review"},{"issue":"1","key":"6030_CR86","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1111\/biom.13392","volume":"77","author":"BD Williamson","year":"2021","unstructured":"Williamson, B. D., Gilbert, P. B., Carone, M., & Simon, N. (2021). Nonparametric variable importance assessment using machine learning techniques. Biometrics, 77(1), 9\u201322. https:\/\/doi.org\/10.1111\/biom.13392","journal-title":"Biometrics"},{"issue":"1","key":"6030_CR79","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1109\/4235.585893","volume":"1","author":"DH Wolpert","year":"1997","unstructured":"Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67\u201382.","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"6030_CR80","doi-asserted-by":"crossref","unstructured":"Wright, M. N., & Ziegler, A. (2017). ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77(1).","DOI":"10.18637\/jss.v077.i01"},{"issue":"17","key":"6030_CR81","doi-asserted-by":"publisher","first-page":"e133","DOI":"10.1093\/nar\/gks461","volume":"40","author":"D Wu","year":"2012","unstructured":"Wu, D., & Smyth, G. K. (2012). Camera: A competitive gene set test accounting for inter-gene correlation. Nucleic Acids Research, 40(17), e133.","journal-title":"Nucleic Acids Research"},{"key":"6030_CR82","unstructured":"Zhang, K., Peters, J., Janzing, D., & Sch\u00f6lkopf, B. (2011). Kernel-based conditional independence test and application in causal discovery. In Proceedings of the International Conference on Uncertainty in Artificial Intelligence (pp. 804\u2013813)."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06030-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06030-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06030-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T13:37:22Z","timestamp":1725543442000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06030-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8]]},"references-count":84,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2021,8]]}},"alternative-id":["6030"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06030-6","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8]]},"assertion":[{"value":"30 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 May 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 June 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 August 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}