{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:52:44Z","timestamp":1791323564572,"version":"4.3.1"},"reference-count":79,"publisher":"Institute of Mathematical Statistics","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Ann. Appl. Stat."],"published-print":{"date-parts":[[2026,6,1]]},"abstract":"Risk of suicide attempt varies over time. Understanding the importance of risk factors measured at a mental health visit can help clinicians evaluate future risk and provide appropriate care during the visit. In prediction settings where data are collected over time, such as in mental health care, it is often of interest to understand both the importance of variables for predicting the response at each time point and the importance summarized over the time series. Building on recent advances in estimation and inference for variable importance measures, we define summaries of variable importance trajectories and corresponding estimators. Under common regularity conditions, the same approaches for inference can be applied to these measures regardless of the choice of the algorithm(s) used to estimate the prediction function under standard convergence conditions. We propose a nonparametric efficient estimation and inference procedure as well as a null hypothesis testing procedure that are valid even when complex machine learning tools are used for prediction. Through simulations we demonstrate that our proposed procedures have good operating characteristics. We use these approaches to analyze electronic health records data from two large health systems to investigate the longitudinal importance of risk factors (i.e., the importance of risk factors over time) for suicide attempt to inform future suicide prevention research and clinical workflow.<\/jats:p>","DOI":"10.1214\/26-aoas2186","type":"journal-article","created":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T13:58:44Z","timestamp":1782395924000},"page":"1340-1363","source":"Crossref","is-referenced-by-count":1,"title":["Inference on summaries of a model-agnostic longitudinal variable importance trajectory with application to suicide prevention"],"prefix":"10.1214","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7024-548X","authenticated-orcid":false,"given":"Brian D","family":"Williamson","sequence":"first","affiliation":[{"name":"Biostatistics Division, Kaiser Permanente Washington Health Research Institute"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7225-3977","authenticated-orcid":false,"given":"Erica E. M.","family":"Moodie","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, Biostatistics, and Occupational Health, McGill University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2859-5467","authenticated-orcid":false,"given":"Gregory E.","family":"Simon","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Sciences, University of Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9531-914X","authenticated-orcid":false,"given":"Rebecca C.","family":"Rossom","sequence":"additional","affiliation":[{"name":"Division of Research, HealthPartners Institute"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7918-601X","authenticated-orcid":false,"given":"Susan M.","family":"Shortreed","sequence":"additional","affiliation":[{"name":"Biostatistics Division, Kaiser Permanente Washington Health Research Institute"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"108","reference":[{"key":"34","doi-asserted-by":"publisher","unstructured":"Lei, J.<\/small>, G\u2019Sell, M.<\/small>, Rinaldo, A.<\/small>, Tibshirani, R. J.<\/small> and Wasserman, L.<\/small> (2018). Distribution-free predictive inference for regression. J. Amer. Statist. Assoc.<\/i> 113<\/b> 1094\u20131111.","DOI":"10.1080\/01621459.2017.1307116"},{"key":"9","doi-asserted-by":"publisher","unstructured":"Chen, S.<\/small>, Tian, L.<\/small>, Cai, T.<\/small> and Yu, M.<\/small> (2017). A general statistical framework for subgroup identification and comparative treatment scoring. Biometrics<\/i> 73<\/b> 1199\u20131209.","DOI":"10.1111\/biom.12676"},{"key":"6","doi-asserted-by":"crossref","unstructured":"Breiman, L.<\/small> (2001). Random forests. Mach. Learn.<\/i> 45<\/b> 5\u201332.","DOI":"10.1023\/A:1010933404324"},{"key":"66","doi-asserted-by":"publisher","unstructured":"VAN DER LAAN, M. J.<\/small>, POLLEY, E. C.<\/small> and HUBBARD, A. E.<\/small> (2007). Super learner. Stat. Appl. Genet. Mol. Biol.<\/i> 6<\/b> Art. 25, 23.","DOI":"10.2202\/1544-6115.1309"},{"key":"74","doi-asserted-by":"publisher","unstructured":"WILLIAMSON, B. D.<\/small>, GILBERT, P. B.<\/small>, CARONE, M.<\/small> and SIMON, N.<\/small> (2021). Nonparametric variable importance assessment using machine learning techniques. Biometrics<\/i> 77<\/b> 9\u201322.","DOI":"10.1111\/biom.13392"},{"key":"27","doi-asserted-by":"publisher","unstructured":"ISHWARAN, H.<\/small>, KOGALUR, U. B.<\/small>, BLACKSTONE, E. H.<\/small> and LAUER, M. S.<\/small> (2008). Random survival forests. Ann. Appl. Stat.<\/i> 2<\/b> 841\u2013860.","DOI":"10.1214\/08-AOAS169"},{"key":"14","doi-asserted-by":"publisher","unstructured":"FAN, A.<\/small>, LU, W.<\/small> and SONG, R.<\/small> (2016). Sequential advantage selection for optimal treatment regime. Ann. Appl. Stat.<\/i> 10<\/b> 32\u201353.","DOI":"10.1214\/15-AOAS849"},{"key":"20","doi-asserted-by":"publisher","unstructured":"GUNTER, L.<\/small>, ZHU, J.<\/small> and MURPHY, S. A.<\/small> (2011). Variable selection for qualitative interactions. Stat. Methodol.<\/i> 8<\/b> 42\u201355.","DOI":"10.1016\/j.stamet.2009.05.003"},{"key":"48","doi-asserted-by":"publisher","unstructured":"ROBINS, J.<\/small>, ORELLANA, L.<\/small> and ROTNITZKY, A.<\/small> (2008). Estimation and extrapolation of optimal treatment and testing strategies. Stat. Med.<\/i> 27<\/b> 4678\u20134721.","DOI":"10.1002\/sim.3301"},{"key":"7","doi-asserted-by":"crossref","unstructured":"BRESLOW, N. E.<\/small> and CLAYTON, D. G.<\/small> (1993). Approximate inference in generalized linear mixed models. J. Amer. Statist. Assoc.<\/i> 88<\/b> 9\u201325.","DOI":"10.1080\/01621459.1993.10594284"},{"key":"15","doi-asserted-by":"publisher","unstructured":"FIELD, C. A.<\/small> and WELSH, A. H.<\/small> (2007). Bootstrapping clustered data. J. R. Stat. Soc. Ser. B. Stat. Methodol.<\/i> 69<\/b> 369\u2013390.","DOI":"10.1111\/j.1467-9868.2007.00593.x"},{"key":"42","doi-asserted-by":"publisher","unstructured":"MURDOCH, W. J.<\/small>, SINGH, C.<\/small>, KUMBIER, K.<\/small>, ABBASI-ASL, R.<\/small> and YU, B.<\/small> (2019). Definitions, methods, and applications in interpretable machine learning. Proc. Natl. Acad. Sci. USA<\/i> 116<\/b> 22071\u201322080.","DOI":"10.1073\/pnas.1900654116"},{"key":"4","doi-asserted-by":"publisher","unstructured":"BIAN, Z.<\/small>, MOODIE, E. E. M.<\/small>, SHORTREED, S. M.<\/small> and BHATNAGAR, S.<\/small> (2023). Variable selection in regression-based estimation of dynamic treatment regimes. Biometrics<\/i> 79<\/b> 988\u2013999.","DOI":"10.1111\/biom.13608"},{"key":"49","doi-asserted-by":"publisher","unstructured":"Rosenbaum, P. R.<\/small> and Rubin, D. B.<\/small> (1983). The central role of the propensity score in observational studies for causal effects. Biometrika<\/i> 70<\/b> 41\u201355.","DOI":"10.1093\/biomet\/70.1.41"},{"key":"23","unstructured":"Hastie, T. J.<\/small> and Tibshirani, R. J.<\/small> (1990). Generalized Additive Models<\/i>. Monographs on Statistics and Applied Probability<\/i> 43<\/b>. CRC Press, London."},{"key":"17","doi-asserted-by":"publisher","unstructured":"Friedman, J. H.<\/small> (2001). Greedy function approximation: A gradient boosting machine. Ann. Statist.<\/i> 29<\/b> 1189\u20131232.","DOI":"10.1214\/aos\/1013203451"},{"key":"35","doi-asserted-by":"publisher","unstructured":"Liang, K. Y.<\/small> and Zeger, S. L.<\/small> (1986). Longitudinal data analysis using generalized linear models. Biometrika<\/i> 73<\/b> 13\u201322.","DOI":"10.1093\/biomet\/73.1.13"},{"key":"75","doi-asserted-by":"publisher","unstructured":"WILLIAMSON, B. D.<\/small>, GILBERT, P. B.<\/small>, SIMON, N. R.<\/small> and CARONE, M.<\/small> (2023). A general framework for inference on algorithm-agnostic variable importance. J. Amer. Statist. Assoc.<\/i> 118<\/b> 1645\u20131658.","DOI":"10.1080\/01621459.2021.2003200"},{"key":"62","doi-asserted-by":"crossref","unstructured":"TIBSHIRANI, R.<\/small> (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B, Methodol.<\/i> 58<\/b> 267\u2013288.","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"1","unstructured":"ADAMS, R.<\/small>, HAROZ, E. E.<\/small>, REBMAN, P.<\/small>, SUTTLE, R.<\/small>, GROSVENOR, L.<\/small>, BAJAJ, M.<\/small>, DAYAL, R. R.<\/small>, MAGGIO, D.<\/small>, KETTERING, C. L.<\/small> et al. (2024). Developing a suicide risk model for use in the Indian Health Service. npj Mental Health Res.<\/i> 3<\/b> 47."},{"key":"2","unstructured":"BARAK-CORREN, Y.<\/small>, CASTRO, V. M.<\/small>, JAVITT, S.<\/small>, HOFFNAGLE, A. G.<\/small>, DAI, Y.<\/small>, PERLIS, R. H.<\/small>, NOCK, M. K.<\/small>, SMOLLER, J. W.<\/small> and REIS, B. Y.<\/small> (2017). Predicting suicidal behavior from longitudinal electronic health records. Amer. J. Psychiatry<\/i> 174<\/b> 154\u2013162."},{"key":"3","unstructured":"BAYRAMLI, I.<\/small>, CASTRO, V.<\/small>, BARAK-CORREN, Y.<\/small>, MADSEN, E. M.<\/small>, NOCK, M. K.<\/small>, SMOLLER, J. W.<\/small> and REIS, B. Y.<\/small> (2022). Temporally informed random forests for suicide risk prediction. J. Amer. Med. Inform. Assoc.<\/i> 29<\/b> 62\u201371."},{"key":"5","doi-asserted-by":"publisher","unstructured":"BOILEAU, P.<\/small>, QI, N. T.<\/small>, VAN DER LAAN, M. J.<\/small>, DUDOIT, S.<\/small> and LENG, N.<\/small> (2023). A flexible approach for predictive biomarker discovery. Biostatistics<\/i> 24<\/b> 1085\u20131105.","DOI":"10.1093\/biostatistics\/kxac029"},{"key":"8","unstructured":"CHEN, Q.<\/small>, ZHANG-JAMES, Y.<\/small>, BARNETT, E. J.<\/small>, LICHTENSTEIN, P.<\/small>, JOKINEN, J.<\/small>, D\u2019ONOFRIO, B. M.<\/small>, FARAONE, S. V.<\/small>, LARSSON, H.<\/small> and FAZEL, S.<\/small> (2020). Predicting suicide attempt or suicide death following a visit to psychiatric specialty care: A machine learning study using Swedish national registry data. PLoS Med.<\/i> 17<\/b> e1003416."},{"key":"10","doi-asserted-by":"crossref","unstructured":"COLEY, R. Y.<\/small>, JOHNSON, E.<\/small>, SIMON, G. E.<\/small>, CRUZ, M.<\/small> and SHORTREED, S. M.<\/small> (2021). Racial\/ethnic disparities in the performance of prediction models for death by suicide after mental health visits. JAMA Psychiatr.<\/i> 78<\/b> 726\u2013734.","DOI":"10.1001\/jamapsychiatry.2021.0493"},{"key":"11","doi-asserted-by":"crossref","unstructured":"COLEY, R. Y.<\/small>, LIAO, Q.<\/small>, SIMON, N.<\/small> and SHORTREED, S. M.<\/small> (2023). Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: A case study in suicide risk prediction. BMC Med. Res. Methodol.<\/i> 23<\/b> 33.","DOI":"10.1186\/s12874-023-01844-5"},{"key":"12","doi-asserted-by":"publisher","unstructured":"COULOMBE, J.<\/small>, MOODIE, E. E. M.<\/small>, SHORTREED, S. M.<\/small> and RENOUX, C.<\/small> (2023). Estimating individualized treatment rules in longitudinal studies with covariate-driven observation times. Stat. Methods Med. Res.<\/i> 32<\/b> 868\u2013884.","DOI":"10.1177\/09622802231158733"},{"key":"13","doi-asserted-by":"crossref","unstructured":"D\u00cdAZ, I.<\/small>, HUBBARD, A.<\/small>, DECKER, A.<\/small> and COHEN, M.<\/small> (2015). Variable importance and prediction methods for longitudinal problems with missing variables. PLoS ONE<\/i> 10<\/b> e0120031.","DOI":"10.1371\/journal.pone.0120031"},{"key":"16","unstructured":"FISHER, A.<\/small>, RUDIN, C.<\/small> and DOMINICI, F.<\/small> (2019). All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res.<\/i> 20<\/b> Paper No. 177, 81."},{"key":"18","doi-asserted-by":"crossref","unstructured":"GRADUS, J. L.<\/small>, ROSELLINI, A. J.<\/small>, HORV\u00c1TH-PUH\u00d3, E.<\/small>, STREET, A. E.<\/small>, GALATZER-LEVY, I.<\/small>, JIANG, T.<\/small>, LASH, T. L.<\/small> and S\u00d8RENSEN, H. T.<\/small> (2020). Prediction of sex-specific suicide risk using machine learning and single-payer health care registry data from Denmark. JAMA Psychiatr.<\/i> 77<\/b> 25\u201334.","DOI":"10.1001\/jamapsychiatry.2019.2905"},{"key":"19","doi-asserted-by":"publisher","unstructured":"GR\u00d6MPING, U.<\/small> (2015). Variable importance in regression models. Wiley Interdiscip. Rev<\/i>.: Comput. Stat.<\/i> 7<\/b> 137\u2013152.","DOI":"10.1002\/wics.1346"},{"key":"21","doi-asserted-by":"crossref","unstructured":"HAROZ, E. E.<\/small>, GRUBIN, F.<\/small>, GOKLISH, N.<\/small>, PIOCHE, S.<\/small>, CWIK, M.<\/small>, BARLOW, A.<\/small>, WAUGH, E.<\/small>, USHER, J.<\/small>, LENERT, M. C.<\/small> et al. (2021). Designing a clinical decision support tool that leverages machine learning for suicide risk prediction: Development study in partnership with Native American care providers. JMIR Public Health Surveill.<\/i> 7<\/b> e24377.","DOI":"10.2196\/24377"},{"key":"22","doi-asserted-by":"crossref","unstructured":"HAROZ, E. E.<\/small>, REBMAN, P.<\/small>, GOKLISH, N.<\/small>, GARCIA, M.<\/small>, SUTTLE, R.<\/small>, MAGGIO, D.<\/small>, CLATTENBURG, E.<\/small>, MEGA, J.<\/small> and ADAMS, R.<\/small> (2024). Performance of machine learning suicide risk models in an American Indian population. JAMA Netw. Open<\/i> 7<\/b> e2439269\u2013e2439269.","DOI":"10.1001\/jamanetworkopen.2024.39269"},{"key":"24","doi-asserted-by":"publisher","unstructured":"HINES, O. J.<\/small>, DIAZ-ORDAZ, K.<\/small> and VANSTEELANDT, S.<\/small> (2025). Variable importance measures for heterogeneous treatment effects. Biometrics<\/i> 81<\/b> Paper No. ujaf140.","DOI":"10.1093\/biomtc\/ujaf140"},{"key":"25","unstructured":"HOROWITZ, L. M.<\/small>, BRIDGE, J. A.<\/small>, TEACH, S. J.<\/small>, BALLARD, E.<\/small>, KLIMA, J.<\/small>, ROSENSTEIN, D. L.<\/small>, WHARFF, E. A.<\/small>, GINNIS, K.<\/small>, CANNON, E.<\/small> et al. (2012). Ask Suicide-Screening Questions (ASQ): A brief instrument for the pediatric emergency department. JAMA Pediatr.<\/i> 166<\/b> 1170\u20131176."},{"key":"26","unstructured":"HUBBARD, A.<\/small>, D\u00cdAZ MU\u00d1OZ, I.<\/small>, DECKER, A.<\/small>, HOLCOMB, J.<\/small>, SCHREIBER, M.<\/small>, BULGER, E.<\/small> et al. (2013). Time-dependent prediction and evaluation of variable importance using superlearning in high dimensional clinical data. J. Trauma Acute Care Surg.<\/i> 75<\/b> S53\u2013S60."},{"key":"28","unstructured":"JACOBS, D. G.<\/small>, BALDESSARINI, R. J.<\/small>, CONWELL, Y.<\/small>, FAWCETT, J. A.<\/small>, HORTON, L.<\/small>, MELTZER, H.<\/small> et al. (2010). Assessment and treatment of patients with suicidal behaviors. APA Pract. Guidel.<\/i> 1<\/b> 183."},{"key":"29","doi-asserted-by":"crossref","unstructured":"KESSLER, R. C.<\/small>, BOSSARTE, R. M.<\/small>, LUEDTKE, A.<\/small>, ZASLAVSKY, A. M.<\/small> and ZUBIZARRETA, J. R.<\/small> (2020). Suicide prediction models: A critical review of recent research with recommendations for the way forward. Mol. Psychiatry<\/i> 25<\/b> 168\u2013179.","DOI":"10.1038\/s41380-019-0531-0"},{"key":"30","unstructured":"KESSLER, R. C.<\/small>, STEIN, M. B.<\/small>, PETUKHOVA, M. V.<\/small>, BLIESE, P.<\/small>, BOSSARTE, R. M.<\/small>, BROMET, E. J.<\/small>, FULLERTON, C. S.<\/small>, GILMAN, S. E.<\/small>, IVANY, C.<\/small> et al. (2017). Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Mol. Psychiatry<\/i> 22<\/b> 544\u2013551."},{"key":"31","unstructured":"KESSLER, R. C.<\/small>, WARNER, L. C. H.<\/small>, IVANY, L. C.<\/small>, PETUKHOVA, M. V.<\/small>, ROSE, S.<\/small>, BROMET, E. J.<\/small>, BROWN, L. M. III<\/small>, CAI, T.<\/small>, COLPE, L. J.<\/small> et al. (2015). Predicting US Army suicides after hospitalizations with psychiatric diagnoses in the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatr.<\/i> 72<\/b> 49."},{"key":"32","doi-asserted-by":"crossref","unstructured":"KROENKE, K.<\/small>, SPITZER, R.<\/small> and WILLIAMS, J.<\/small> (2001). The PHQ-9: Validity of a brief depression severity measure. J. Gen. Intern. Med.<\/i> 16<\/b> 606\u2013613.","DOI":"10.1046\/j.1525-1497.2001.016009606.x"},{"key":"33","doi-asserted-by":"crossref","unstructured":"KROENKE, K.<\/small>, SPITZER, R. L.<\/small>, WILLIAMS, J. B.<\/small> and L\u00d6WE, B.<\/small> (2010). The patient health questionnaire somatic, anxiety, and depressive symptom scales: A systematic review. Gen. Hosp. Psychiatry<\/i> 32<\/b> 345\u2013359.","DOI":"10.1016\/j.genhosppsych.2010.03.006"},{"key":"36","doi-asserted-by":"crossref","unstructured":"LINK, W.<\/small> and SAUER, J.<\/small> (1997). Estimation of population trajectories from count data. Biometrics<\/i> 53<\/b> 488\u2013497.","DOI":"10.2307\/2533952"},{"key":"37","unstructured":"LUNDBERG, S.<\/small> and LEE, S.-I.<\/small> (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems<\/i> 30<\/b>."},{"key":"38","unstructured":"MA, T.<\/small>, CAI, H.<\/small>, QI, Z.<\/small>, SHI, C.<\/small> and LABER, E.<\/small> (2023). Sequential knockoffs for variable selection in reinforcement learning. arXiv preprint. Available at arXiv:2303.14281<\/a>."},{"key":"39","doi-asserted-by":"crossref","unstructured":"MATARAZZO, B. B.<\/small>, EAGAN, A.<\/small>, LANDES, S. J.<\/small>, MINA, L. K.<\/small> et al. (2023). The Veterans Health Administration REACH VET program: Suicide predictive modeling in practice. Psychiatr. Serv.<\/i> 74<\/b> 206\u2013209.","DOI":"10.1176\/appi.ps.202100629"},{"key":"40","doi-asserted-by":"publisher","unstructured":"MINUS, E.<\/small>, COLEY, R. Y.<\/small>, SHORTREED, S. M.<\/small> and WILLIAMSON, B. D.<\/small> (2025). Behavior of prediction performance metrics with rare events. J. Clin. Epidemiol.<\/i>","DOI":"10.1016\/j.jclinepi.2025.112046"},{"key":"41","doi-asserted-by":"crossref","unstructured":"MUNDT, J.<\/small>, GRIEST, J.<\/small>, JEFFERSON, J.<\/small>, FEDERICO, M.<\/small>, MANN, J.<\/small> and POSNER, K.<\/small> (2013). Prediction of suicidal behavior in clinical research by lifetime suicidal ideation and behavior ascertained by the electronic Columbia-Suicide Severity Rating Scale. J. Clin. Psychiatry<\/i> 74<\/b> 887\u2013893.","DOI":"10.4088\/JCP.13m08398"},{"key":"43","doi-asserted-by":"crossref","unstructured":"NOCK, M. K.<\/small>, MILLNER, A. J.<\/small>, ROSS, E. L.<\/small>, KENNEDY, C. J.<\/small>, AL-SUWAIDI, M.<\/small>, BARAK-CORREN, Y.<\/small>, CASTRO, V. M.<\/small>, CASTRO-RAMIREZ, F.<\/small>, LAURICELLA, T.<\/small> et al. (2022). Prediction of suicide attempts using clinician assessment, patient self-report, and electronic health records. JAMA Netw. Open<\/i> 5<\/b> e2144373\u2013e2144373.","DOI":"10.1001\/jamanetworkopen.2021.44373"},{"key":"44","unstructured":"PAPINI, S.<\/small>, HSIN, H.<\/small>, KIPNIS, P.<\/small>, LIU, V. X.<\/small>, LU, Y.<\/small>, GIRARD, K.<\/small>, STERLING, S. A.<\/small> and ITURRALDE, E. M.<\/small> (2024). Validation of a multivariable model to predict suicide attempt in a mental health intake sample. JAMA Psychiatr.<\/i> 81<\/b> 700\u2013707."},{"key":"45","unstructured":"PAPINI, S.<\/small>, HSIN, H.<\/small>, KIPNIS, P.<\/small>, LIU, V. X.<\/small>, LU, Y.<\/small>, STERLING, S. A.<\/small> and ITURRALDE, E.<\/small> (2023). Performance of a prediction model of suicide attempts across race and ethnicity. JAMA Psychiatr.<\/i> 80<\/b> 399\u2013400."},{"key":"46","unstructured":"PENFOLD, R. B.<\/small>, JOHNSON, E.<\/small>, SHORTREED, S. M.<\/small>, ZIEBELL, R. A.<\/small>, LYNCH, F. L.<\/small>, CLARKE, G. N.<\/small>, COLEMAN, K. J.<\/small>, WAITZFELDER, B. E.<\/small>, BECK, A. L.<\/small> et al. (2021). Predicting suicide attempts and suicide deaths among adolescents following outpatient visits. J. Affective Disorders<\/i> 294<\/b> 39\u201347."},{"key":"47","unstructured":"POSNER, K.<\/small>, BROWN, G. K.<\/small>, STANLEY, B.<\/small>, BRENT, D. A.<\/small>, YERSHOVA, K. V.<\/small>, OQUENDO, M. A.<\/small>, CURRIER, G. W.<\/small>, MELVIN, G. A.<\/small>, GREENHILL, L.<\/small> et al. (2011). The Columbia\u2013Suicide Severity Rating Scale: Initial validity and internal consistency findings from three multisite studies with adolescents and adults. Amer. J. Psychiatry<\/i> 168<\/b> 1266\u20131277."},{"key":"50","unstructured":"ROSSOM, R. C.<\/small>, RICHARDS, J. E.<\/small>, STERLING, S.<\/small>, AHMEDANI, B.<\/small>, BOGGS, J. M.<\/small>, YARBOROUGH, B. J. H.<\/small>, BECK, A.<\/small>, LLOYD, K.<\/small>, FRANK, C.<\/small> et al. (2022). Connecting research and practice: Implementation of suicide prevention strategies in learning health care systems. Psychiatr. Serv.<\/i> 73<\/b> 219\u2013222."},{"key":"51","doi-asserted-by":"crossref","unstructured":"SANDERSON, M.<\/small>, BULLOCH, A. G.<\/small>, WANG, J.<\/small>, WILLIAMS, K. G.<\/small>, WILLIAMSON, T.<\/small> and PATTEN, S. B.<\/small> (2020). Predicting death by suicide following an emergency department visit for parasuicide with administrative health care system data and machine learning. EClinicalMedicine<\/i> 20<\/b>.","DOI":"10.1016\/j.eclinm.2020.100281"},{"key":"52","doi-asserted-by":"crossref","unstructured":"SANDERSON, M.<\/small>, BULLOCH, A. G.<\/small>, WANG, J.<\/small>, WILLIAMSON, T.<\/small> and PATTEN, S. B.<\/small> (2019). Predicting death by suicide using administrative health care system data: Can feedforward neural network models improve upon logistic regression models? J. Affective Disorders<\/i> 257<\/b> 741\u2013747.","DOI":"10.1016\/j.jad.2019.07.063"},{"key":"53","unstructured":"SHAW, J. L.<\/small>, BEANS, J. A.<\/small>, NOONAN, C.<\/small>, SMITH, J. J.<\/small>, MOSLEY, M.<\/small>, LILLIE, K. M.<\/small>, AVEY, J. P.<\/small>, ZIEBELL, R.<\/small> and SIMON, G.<\/small> (2022). Validating a predictive algorithm for suicide risk with Alaska Native populations. Suicide Life-Threat. Behav.<\/i> 52<\/b> 696\u2013704."},{"key":"54","unstructured":"SHORTREED, S.<\/small>, WALKER, R.<\/small>, JOHNSON, E.<\/small>, WELLMAN, R.<\/small>, CRUZ, M.<\/small>, ZIEBELL, R.<\/small> et al. (2023). Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction. npj Digit. Med.<\/i> 6<\/b> 47."},{"key":"55","doi-asserted-by":"crossref","unstructured":"SIMON, G.<\/small>, JOHNSON, E.<\/small>, LAWRENCE, J.<\/small>, ROSSOM, R.<\/small>, AHMEDANI, B.<\/small>, LYNCH, F.<\/small> et al. (2018). Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Amer. J. Psychiatry<\/i> 175<\/b> 951\u2013960.","DOI":"10.1176\/appi.ajp.2018.17101167"},{"key":"56","doi-asserted-by":"crossref","unstructured":"SIMON, G.<\/small>, RUTTER, C.<\/small>, PETERSON, D.<\/small>, OLIVER, M.<\/small>, WHITESIDE, U.<\/small>, OPERSKALSKI, B.<\/small> and LUDMAN, E.<\/small> (2013). Does response on the PHQ-9 depression questionnaire predict subsequent suicide attempt or suicide death? Psychiatr. Serv.<\/i> 64<\/b> 1195\u20131202.","DOI":"10.1176\/appi.ps.201200587"},{"key":"57","unstructured":"SIMON, G.<\/small>, YARBOROUGH, B.<\/small>, ROSSON, R.<\/small>, LAWRENCE, J.<\/small>, LYNCH, F.<\/small>, WAITZFELDER, B.<\/small>, AHMEDANI, B.<\/small> and SHORTREED, S.<\/small> (2019a). Self-reported suicidal ideation predicts suicidal behavior in out-patient receiving psychotic disorder diagnosis. Psychiatr. Serv.<\/i> 70<\/b> 176\u201383."},{"key":"58","unstructured":"SIMON, G. E.<\/small>, COLEMAN, K. J.<\/small>, ROSSOM, R. C.<\/small>, BECK, A.<\/small>, OLIVER, M.<\/small>, JOHNSON, E.<\/small>, WHITESIDE, U.<\/small>, OPERSKALSKI, B.<\/small>, PENFOLD, R. B.<\/small> et al. (2016). Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J. Clin. Psychiatry<\/i> 77<\/b> 20461."},{"key":"59","unstructured":"SIMON, G. E.<\/small>, CRUZ, M.<\/small>, SHORTREED, S. M.<\/small>, STERLING, S. A.<\/small>, COLEMAN, K. J.<\/small>, AHMEDANI, B. K.<\/small>, YASEEN, Z. S.<\/small> and MOSHOLDER, A. D.<\/small> (2024). Stability of suicide risk prediction models during changes in health care delivery. Psychiatr. Serv.<\/i> 75<\/b> 139\u2013147."},{"key":"60","unstructured":"SIMON, G. E.<\/small>, SHORTREED, S. M.<\/small>, JOHNSON, E.<\/small>, BECK, A.<\/small>, COLEMAN, K. J.<\/small>, ROSSOM, R. C.<\/small>, WHITESIDE, U. S.<\/small>, OPERSKALSKI, B. H.<\/small> and PENFOLD, R. B.<\/small> (2017). Between-visit changes in suicidal ideation and risk of subsequent suicide attempt. Depress. Anxiety<\/i> 34<\/b> 794\u2013800."},{"key":"61","unstructured":"SIMON, G. E.<\/small>, SHORTREED, S. M.<\/small>, JOHNSON, E.<\/small>, ROSSOM, R. C.<\/small>, LYNCH, F. L.<\/small>, ZIEBELL, R.<\/small> and PENFOLD, R. B.<\/small> (2019b). What health records data are required for accurate prediction of suicidal behavior? J. Amer. Med. Inform. Assoc.<\/i> 26<\/b> 1458\u20131465."},{"key":"63","unstructured":"TSUI, F. R.<\/small>, SHI, L.<\/small>, RUIZ, V.<\/small>, RYAN, N. D.<\/small>, BIERNESSER, C.<\/small>, IYENGAR, S.<\/small>, WALSH, C. G.<\/small> and BRENT, D. A.<\/small> (2021). Natural language processing and machine learning of electronic health records for prediction of first-time suicide attempts. JAMIA Open<\/i> 4<\/b> ooab011."},{"key":"65","doi-asserted-by":"publisher","unstructured":"VAN DER LAAN, M. J.<\/small> and LUEDTKE, A. R.<\/small> (2015). Targeted learning of the mean outcome under an optimal dynamic treatment rule. J. Causal Inference<\/i> 3<\/b> 61\u201395.","DOI":"10.1515\/jci-2013-0022"},{"key":"67","doi-asserted-by":"crossref","unstructured":"VOGT, D.<\/small>, ROSELLINI, A. J.<\/small>, BOROWSKI, S.<\/small>, STREET, A. E.<\/small>, O\u2019BRIEN, R. W.<\/small> and TOMOYASU, N.<\/small> (2024). How well can US military veterans\u2019 suicidal ideation be predicted from static and change-based indicators of their psychosocial well-being as they adapt to civilian life? Soc. Psychiatry Psychiatr. Epidemiol.<\/i> 59<\/b> 261\u2013271.","DOI":"10.1007\/s00127-023-02511-2"},{"key":"68","unstructured":"WALKER, R. L.<\/small>, SHORTREED, S. M.<\/small>, ZIEBELL, R. A.<\/small>, JOHNSON, E.<\/small>, BOGGS, J. M.<\/small>, LYNCH, F. L.<\/small>, DAIDA, Y. G.<\/small>, AHMEDANI, B. K.<\/small>, ROSSOM, R.<\/small> et al. (2021). Evaluation of electronic health record-based suicide risk prediction models on contemporary data. Appl. Clin. Inform.<\/i> 12<\/b> 778\u2013787."},{"key":"69","doi-asserted-by":"crossref","unstructured":"WALSH, C. G.<\/small>, RIBEIRO, J. D.<\/small> and FRANKLIN, J. C.<\/small> (2017). Predicting risk of suicide attempts over time through machine learning. Clin. Psychol. Sci.<\/i> 5<\/b> 457\u2013469.","DOI":"10.1177\/2167702617691560"},{"key":"70","doi-asserted-by":"crossref","unstructured":"WALSH, C. G.<\/small>, RIBEIRO, J. D.<\/small> and FRANKLIN, J. C.<\/small> (2018). Predicting suicide attempts in adolescents with longitudinal clinical data and machine learning. J. Child Psychol. Psychiatry<\/i> 59<\/b> 1261\u20131270.","DOI":"10.1111\/jcpp.12916"},{"key":"71","doi-asserted-by":"crossref","unstructured":"WEI, P.<\/small>, LU, Z.<\/small> and SONG, J.<\/small> (2015). Variable importance analysis: A comprehensive review. Reliab. Eng. Syst. Saf.<\/i> 142<\/b> 399\u2013432.","DOI":"10.1016\/j.ress.2015.05.018"},{"key":"72","unstructured":"WILLIAMSON, B.<\/small> (2023). lvimp: Perform Inference on Summaries of Longitudinal Algorithm-Agnostic Variable Importance R package version 0.0.0.9000."},{"key":"73","unstructured":"WILLIAMSON, B.<\/small> and FENG, J.<\/small> (2020). Efficient nonparametric statistical inference on population feature importance using Shapley values. In Proceedings of the<\/i> 37th International Conference on Machine Learning<\/i>. Proceedings of Machine Learning Research<\/i> 119<\/b> 10282\u201310291."},{"key":"76","doi-asserted-by":"crossref","unstructured":"WILLIAMSON, B. D.<\/small>, MOODIE, E. E.<\/small>, SIMON, G. E.<\/small>, ROSSOM, R. C.<\/small> and SHORTREED, S. M.<\/small> (2026). Supplement to \u201cInference on summaries of a model-agnostic longitudinal variable importance trajectory with application to suicide prevention.\u201d https:\/\/doi.org\/10.1214\/26-AOAS2186SUPPA<\/a>, https:\/\/doi.org\/10.1214\/26-AOAS2186SUPPB<\/a>","DOI":"10.1214\/26-AOAS2186SUPPB"},{"key":"77","unstructured":"WOLOCK, C. J.<\/small>, WILLIAMSON, B. D.<\/small>, SHORTREED, S. M.<\/small>, SIMON, G. E.<\/small>, COLEMAN, K. J.<\/small> et al. (2024). Importance of variables from different time frames for predicting self-harm using health system data. medRxiv 2024-04."},{"key":"78","doi-asserted-by":"publisher","unstructured":"WU, J.<\/small>, GALANTER, N.<\/small>, SHORTREED, S. M.<\/small> and MOODIE, E. E. M.<\/small> (2022). Ranking tailoring variables for constructing individualized treatment rules: An application to schizophrenia. J. R. Stat. Soc<\/i>., Ser. C, Appl. Stat.<\/i> 71<\/b> 309\u2013330.","DOI":"10.1111\/rssc.12533"},{"key":"79","unstructured":"ZHENG, L.<\/small>, WANG, O.<\/small>, HAO, S.<\/small>, YE, C.<\/small>, LIU, M.<\/small>, XIA, M.<\/small>, SABO, A. N.<\/small>, MARKOVIC, L.<\/small>, STEARNS, F.<\/small> et al. (2020). Development of an early-warning system for high-risk patients for suicide attempt using deep learning and electronic health records. Transl. Psychiatry<\/i> 10<\/b> 72."},{"key":"64","doi-asserted-by":"publisher","unstructured":"VAN DER LAAN, M. J.<\/small> (2006). Statistical inference for variable importance. Int. J. Biostat.<\/i> 2<\/b> Art. 2, 33.","DOI":"10.2202\/1557-4679.1008"}],"container-title":["The Annals of Applied Statistics"],"original-title":[],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:21:00Z","timestamp":1791321660000},"score":1,"resource":{"primary":{"URL":"https:\/\/projecteuclid.org\/journals\/annals-of-applied-statistics\/volume-20\/issue-2\/Inference-on-summaries-of-a-model-agnostic-longitudinal-variable-importance\/10.1214\/26-AOAS2186.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,1]]},"references-count":79,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,6,1]]}},"URL":"https:\/\/doi.org\/10.1214\/26-aoas2186","relation":{},"ISSN":["1932-6157","1941-7330"],"issn-type":[{"value":"1932-6157","type":"print"},{"value":"1941-7330","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,1]]}}}