{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T23:36:20Z","timestamp":1787960180879,"version":"build-2784847793"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2024,11,4]],"date-time":"2024-11-04T00:00:00Z","timestamp":1730678400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,3,16]]},"abstract":"<jats:title>Summary<\/jats:title>\n                  <jats:p>Given a collection of features available for inclusion in a predictive model, it may be of interest to quantify the relative importance of a subset of features for the prediction task at hand. For example, in HIV vaccine trials, participant baseline characteristics are used to predict the probability of HIV acquisition over the intended follow-up period, and investigators may wish to understand how much certain types of predictors, such as behavioural factors, contribute to overall predictiveness. Time-to-event outcomes such as time to HIV acquisition are often subject to right censoring, and existing methods for assessing variable importance are typically not intended to be used in this setting. We describe a broad class of algorithm-agnostic variable importance measures for prediction in the context of survival data. We propose a nonparametric efficient estimation procedure that incorporates flexible learning of nuisance parameters, yields asymptotically valid inference and enjoys double robustness. We assess the performance of our proposed procedure via numerical simulations and analyse data from the HVTN 702 vaccine trial to inform enrolment strategies for future HIV vaccine trials.<\/jats:p>","DOI":"10.1093\/biomet\/asae061","type":"journal-article","created":{"date-parts":[[2024,10,31]],"date-time":"2024-10-31T00:18:35Z","timestamp":1730333915000},"source":"Crossref","is-referenced-by-count":5,"title":["Assessing variable importance in survival analysis using machine learning"],"prefix":"10.1093","volume":"112","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3527-1102","authenticated-orcid":false,"given":"C J","family":"Wolock","sequence":"first","affiliation":[{"name":"Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania , 432 Guardian Drive , Philadelphia, Pennsylvania 19104,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2662-9427","authenticated-orcid":false,"given":"P B","family":"Gilbert","sequence":"additional","affiliation":[{"name":"Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center , 1100 Fairview Avenue North , PO Box 19024, Seattle, Washington 98109,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"N","family":"Simon","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, University of Washington , 3980 15th Avenue NE , Seattle, Washington 98195,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M","family":"Carone","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, University of Washington , 3980 15th Avenue NE , Seattle, Washington 98195,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,11,4]]},"reference":[{"key":"2025110405014577000_asae061-B1","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1097\/QAI.0000000000000974","article-title":"An empiric HIV risk scoring tool to predict HIV-1 acquisition in African women","volume":"72","author":"Balkus","year":"2016","journal-title":"J. Acquir. Immune Defic. Syndr"},{"key":"2025110405014577000_asae061-B2","volume-title":"Efficient and Adaptive Estimation for Semiparametric Models","author":"Bickel","year":"1998"},{"key":"2025110405014577000_asae061-B3","author":"Boileau","year":"2024"},{"key":"2025110405014577000_asae061-B4","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn"},{"key":"2025110405014577000_asae061-B5","doi-asserted-by":"crossref","first-page":"2359","DOI":"10.1177\/0962280216680245","article-title":"Use of the concordance index for predictors of censored survival data","volume":"27","author":"Brentnall","year":"2018","journal-title":"Statist. Meth. Med. Res"},{"key":"2025110405014577000_asae061-B6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"78","author":"Brier","year":"1950","journal-title":"Monthly Weather Rev"},{"key":"2025110405014577000_asae061-B7","doi-asserted-by":"crossref","first-page":"3474","DOI":"10.1002\/sim.2299","article-title":"Estimation of time-dependent area under the ROC curve for long-term risk prediction","volume":"25","author":"Chambless","year":"2006","journal-title":"Statist. Med"},{"key":"2025110405014577000_asae061-B8","doi-asserted-by":"crossref","first-page":"C1","DOI":"10.1111\/ectj.12097","article-title":"Double\/debiased machine learning for treatment and structural parameters","volume":"21","author":"Chernozhukov","year":"2018","journal-title":"Economet. J"},{"key":"2025110405014577000_asae061-B9","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.2517-6161.1972.tb00899.x","article-title":"Regression models and life-tables","volume":"34","author":"Cox","year":"1972","journal-title":"J. R. Statist. Soc. B"},{"key":"2025110405014577000_asae061-B10","doi-asserted-by":"crossref","first-page":"1898","DOI":"10.1109\/TNNLS.2022.3185742","article-title":"Significance tests of feature relevance for a black-box learner","volume":"35","author":"Dai","year":"2022","journal-title":"IEEE Trans. Neural Networks Learn. Syst"},{"key":"2025110405014577000_asae061-B11","first-page":"831","volume-title":"Fifth Berkeley Symposium on Mathematical Statistics and Probability","author":"Efron","year":"1967"},{"key":"2025110405014577000_asae061-B12","first-page":"1","article-title":"All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously","volume":"20","author":"Fisher","year":"2019","journal-title":"J. Mach. Learn. Res."},{"key":"2025110405014577000_asae061-B13","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1002\/bimj.200610301","article-title":"Consistent estimation of the expected Brier score in general survival models with right-censored event times","volume":"48","author":"Gerds","year":"2006","journal-title":"Biomet. J"},{"key":"2025110405014577000_asae061-B14","doi-asserted-by":"crossref","first-page":"1501","DOI":"10.1214\/aos\/1176347865","article-title":"A survey of product-integration with a view toward application in survival analysis","volume":"18","author":"Gill","year":"1990","journal-title":"Ann. Statist"},{"key":"2025110405014577000_asae061-B15","doi-asserted-by":"crossref","first-page":"965","DOI":"10.1093\/biomet\/92.4.965","article-title":"Concordance probability and discriminatory power in proportional hazards regression","volume":"92","author":"Gonen","year":"2005","journal-title":"Biometrika"},{"key":"2025110405014577000_asae061-B16","doi-asserted-by":"crossref","first-page":"1089","DOI":"10.1056\/NEJMoa2031499","article-title":"Vaccine efficacy of ALVAC-HIV and bivalent subtype C gp120\u2013MF59 in adults","volume":"384","author":"Gray","year":"2021","journal-title":"New Engl. J. Med"},{"key":"2025110405014577000_asae061-B17","doi-asserted-by":"crossref","first-page":"2543","DOI":"10.1001\/jama.1982.03320430047030","article-title":"Evaluating the yield of medical tests","volume":"247","author":"Harrell","year":"1982","journal-title":"J. Am. Med. Assoc"},{"key":"2025110405014577000_asae061-B18","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1111\/j.0006-341X.2005.030814.x","article-title":"Survival model predictive accuracy and ROC curves","volume":"61","author":"Heagerty","year":"2005","journal-title":"Biometrics"},{"key":"2025110405014577000_asae061-B19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11222-021-10057-z","article-title":"Unrestricted permutation forces extrapolation: Variable importance requires at least one more model, or there is no free variable importance","volume":"31","author":"Hooker","year":"2021","journal-title":"Statist. Comp"},{"key":"2025110405014577000_asae061-B20","author":"Hudson","year":"2023"},{"key":"2025110405014577000_asae061-B21","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1002\/cjs.10046","article-title":"Estimation methods for time-dependent AUC models with survival data","volume":"38","author":"Hung","year":"2010","journal-title":"Can. J. Statist"},{"key":"2025110405014577000_asae061-B22","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1214\/08-AOAS169","article-title":"Random survival forests","volume":"2","author":"Ishwaran","year":"2008","journal-title":"Ann. Appl. Statist."},{"key":"2025110405014577000_asae061-B23","doi-asserted-by":"crossref","first-page":"558","DOI":"10.1002\/sim.7803","article-title":"Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival","volume":"38","author":"Ishwaran","year":"2019","journal-title":"Statist. Med"},{"key":"2025110405014577000_asae061-B24","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1002\/sim.4780090503","article-title":"Measures of explained variation for survival data","volume":"9","author":"Korn","year":"1990","journal-title":"Statist. Med."},{"key":"2025110405014577000_asae061-B25","doi-asserted-by":"crossref","first-page":"106164","DOI":"10.1016\/j.knosys.2020.106164","article-title":"SurvLIME: A method for explaining machine learning survival models","volume":"203","author":"Kovalev","year":"2020","journal-title":"Knowledge-Based Syst"},{"key":"2025110405014577000_asae061-B26","doi-asserted-by":"crossref","first-page":"110234","DOI":"10.1016\/j.knosys.2022.110234","article-title":"SurvSHAP (t): Time-dependent explanations of machine learning survival models","volume":"262","author":"Krzyzi\u0144ski","year":"2023","journal-title":"Knowledge-Based Syst"},{"key":"2025110405014577000_asae061-B27","author":"Lundborg","year":"2024"},{"key":"2025110405014577000_asae061-B28","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1097\/OLQ.0b013e3181a9cc41","article-title":"Prediction of HIV acquisition among men who have sex with men","volume":"36","author":"Menza","year":"2009","journal-title":"Sex. Transm. Dis"},{"key":"2025110405014577000_asae061-B29","volume-title":"R: A Language and Environment for Statistical Computing","author":"R Development Core Team","year":"2025"},{"key":"2025110405014577000_asae061-B30","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1214\/aos\/1176345334","article-title":"Influence functions for censored data","volume":"9","author":"Reid","year":"1981","journal-title":"Ann. Statist."},{"key":"2025110405014577000_asae061-B31","doi-asserted-by":"crossref","first-page":"4","DOI":"10.2202\/1557-4679.1052","article-title":"A doubly robust censoring unbiased transformation","volume":"3","author":"Rubin","year":"2007","journal-title":"Int. J. Biostatist"},{"key":"2025110405014577000_asae061-B32","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1111\/j.0006-341X.2000.00249.x","article-title":"Predictive accuracy and explained variation in Cox regression","volume":"56","author":"Schemper","year":"2000","journal-title":"Biometrics"},{"key":"2025110405014577000_asae061-B33","doi-asserted-by":"crossref","DOI":"10.1002\/9780470316481","volume-title":"Approximation Theorems of Mathematical Statistics","author":"Serfling","year":"1980"},{"key":"2025110405014577000_asae061-B34","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1097\/QAI.0b013e318256b2f6","article-title":"Development of a clinical screening index predictive of incident HIV infection among men who have sex with men in the United States","volume":"60","author":"Smith","year":"2012","journal-title":"J. Acquir. Immune Defic. Syndr"},{"key":"2025110405014577000_asae061-B35","first-page":"947","article-title":"A semiparametric approach for the covariate specific ROC curve with survival outcome","volume":"18","author":"Song","year":"2008","journal-title":"Statist. Sinica"},{"key":"2025110405014577000_asae061-B36","doi-asserted-by":"crossref","first-page":"btad723","DOI":"10.1093\/bioinformatics\/btad723","article-title":"survex: An R package for explaining machine learning survival models","volume":"39","author":"Spytek","year":"2023","journal-title":"Bioinformatics"},{"key":"2025110405014577000_asae061-B37","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1002\/sim.4154","article-title":"On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data","volume":"30","author":"Uno","year":"2011","journal-title":"Statist. Med"},{"key":"2025110405014577000_asae061-B38","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1198\/016214507000000149","article-title":"Evaluating prediction rules for $ t $-year survivors with censored regression models","volume":"102","author":"Uno","year":"2007","journal-title":"J. Am. Statist. Assoc"},{"key":"2025110405014577000_asae061-B39","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-21700-0","volume-title":"Unified Methods for Censored Longitudinal Data and Causality","author":"van der Laan","year":"2003"},{"key":"2025110405014577000_asae061-B40","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-9782-1","volume-title":"Targeted Learning: Causal Inference for Observational and Experimental Data","author":"van der Laan","year":"2011"},{"key":"2025110405014577000_asae061-B41","first-page":"1","article-title":"Decorrelated variable importance","volume":"25","author":"Verdinelli","year":"2024","journal-title":"J. Mach. Learn. Res"},{"key":"2025110405014577000_asae061-B42","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1093\/biomet\/asaa027","article-title":"Combining $ p $-values via averaging","volume":"107","author":"Vovk","year":"2020","journal-title":"Biometrika"},{"key":"2025110405014577000_asae061-B43","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/s10461-017-1785-7","article-title":"A simple risk prediction algorithm for HIV transmission: Results from HIV prevention trials in KwaZulu Natal, South Africa (2002\u20132012)","volume":"22","author":"Wand","year":"2018","journal-title":"AIDS Behav"},{"key":"2025110405014577000_asae061-B44","doi-asserted-by":"crossref","first-page":"1541","DOI":"10.1080\/01621459.2023.2205060","article-title":"Inference for treatment-specific survival curves using machine learning","volume":"119","author":"Westling","year":"2024","journal-title":"J. Am. Statist. Assoc"},{"key":"2025110405014577000_asae061-B45","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1111\/biom.13389","article-title":"Rejoinder to \u2018Nonparametric variable importance assessment using machine learning techniques\u2019","volume":"77","author":"Williamson","year":"2021","journal-title":"Biometrics"},{"key":"2025110405014577000_asae061-B46","doi-asserted-by":"crossref","first-page":"1645","DOI":"10.1080\/01621459.2021.2003200","article-title":"A general framework for inference on algorithm-agnostic variable importance","volume":"118","author":"Williamson","year":"2023","journal-title":"J. Am. Statist. Assoc"},{"key":"2025110405014577000_asae061-B47","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1080\/10618600.2024.2304070","article-title":"A framework for leveraging machine learning tools to estimate personalized survival curves","volume":"33","author":"Wolock","year":"2024","journal-title":"J. Comp. Graph. Statist"},{"key":"2025110405014577000_asae061-B48","author":"Zhang","year":"2022"},{"key":"2025110405014577000_asae061-B49","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1007\/978-1-4419-9782-1_27","volume-title":"Targeted Learning: Causal Inference for Observational Data","author":"Zheng","year":"2011"}],"container-title":["Biometrika"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/biomet\/advance-article-pdf\/doi\/10.1093\/biomet\/asae061\/60413755\/asae061.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/112\/2\/asae061\/60413755\/asae061.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/biomet\/article-pdf\/112\/2\/asae061\/60413755\/asae061.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T10:02:01Z","timestamp":1762250521000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/biomet\/article\/doi\/10.1093\/biomet\/asae061\/7874959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,4]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,3,16]]}},"URL":"https:\/\/doi.org\/10.1093\/biomet\/asae061","relation":{},"ISSN":["1464-3510"],"issn-type":[{"value":"1464-3510","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025]]},"published":{"date-parts":[[2024,11,4]]},"article-number":"asae061"}}