{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T19:28:33Z","timestamp":1768678113091,"version":"3.49.0"},"reference-count":25,"publisher":"Georg Thieme Verlag KG","issue":"04","funder":[{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"crossref","award":["U19 MH092201"],"award-info":[{"award-number":["U19 MH092201"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"crossref","award":["U19 MH121738"],"award-info":[{"award-number":["U19 MH121738"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Background\u2003Suicide risk prediction models have been developed by using information from patients' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data.<\/jats:p><jats:p>Objectives\u2003A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models' performance with contemporary data (2014\u20132017) from these systems.<\/jats:p><jats:p>Methods\u2003We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive\/negative predictive value.<\/jats:p><jats:p>Results\u2003Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860\u20130.864) and 0.864 (95% CI: 0.860\u20130.869) for suicide attempt, and 0.806 (95% CI: 0.790\u20130.822) and 0.804 (95% CI: 0.782\u20130.829) for suicide death.<\/jats:p><jats:p>Conclusion\u2003Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today.<\/jats:p>","DOI":"10.1055\/s-0041-1733908","type":"journal-article","created":{"date-parts":[[2021,8,18]],"date-time":"2021-08-18T23:50:23Z","timestamp":1629330623000},"page":"778-787","source":"Crossref","is-referenced-by-count":13,"title":["Evaluation of Electronic Health Record-Based Suicide Risk Prediction Models on Contemporary Data"],"prefix":"10.1055","volume":"12","author":[{"given":"Rod L.","family":"Walker","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States"}]},{"given":"Susan M.","family":"Shortreed","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States"}]},{"given":"Rebecca A.","family":"Ziebell","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States"}]},{"given":"Eric","family":"Johnson","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States"}]},{"given":"Jennifer M.","family":"Boggs","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Colorado, Institute for Health Research, Aurora, Colorado, United States"}]},{"given":"Frances L.","family":"Lynch","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Northwest, Center for Health Research, Portland, Oregon, United States"}]},{"given":"Yihe G.","family":"Daida","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Hawaii, Center for Integrated Health Care Research, Honolulu, Hawaii, United States"}]},{"given":"Brian K.","family":"Ahmedani","sequence":"additional","affiliation":[{"name":"Henry Ford Health System, Center for Health Policy & Health Services Research, Detroit, Michigan, United States"}]},{"given":"Rebecca","family":"Rossom","sequence":"additional","affiliation":[{"name":"Department of Research, HealthPartners Institute, Minneapolis, Minnesota, United States"}]},{"given":"Karen J.","family":"Coleman","sequence":"additional","affiliation":[{"name":"Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, California, United States"}]},{"given":"Gregory E.","family":"Simon","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States"}]}],"member":"194","published-online":{"date-parts":[[2021,8,18]]},"reference":[{"issue":"02","key":"ref1","doi-asserted-by":"crossref","first-page":"221","DOI":"10.4088\/JCP.15m09776","article-title":"Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice","volume":"77","author":"G E Simon","year":"2016","journal-title":"J Clin Psychiatry"},{"issue":"02","key":"ref2","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1176\/appi.ajp.2016.16010077","article-title":"Predicting suicidal behavior from longitudinal electronic health records","volume":"174","author":"Y Barak-Corren","year":"2017","journal-title":"Am J Psychiatry"},{"key":"ref3","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.jad.2018.01.019","article-title":"Ten-year prediction of suicide death using Cox regression and machine learning in a nationwide retrospective cohort study in South Korea","volume":"231","author":"S B Choi","year":"2018","journal-title":"J Affect Disord"},{"issue":"03","key":"ref4","doi-asserted-by":"crossref","first-page":"e1575","DOI":"10.1002\/mpr.1575","article-title":"Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration","volume":"26","author":"R C Kessler","year":"2017","journal-title":"Int J Methods Psychiatr Res"},{"issue":"10","key":"ref5","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1176\/appi.ajp.2018.17101167","article-title":"Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records","volume":"175","author":"G E Simon","year":"2018","journal-title":"Am J Psychiatry"},{"issue":"03","key":"ref6","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1177\/2167702617691560","article-title":"Predicting risk of suicide attempts over time through machine learning","volume":"5","author":"C G Walsh","year":"2017","journal-title":"Clin Psychol Sci"},{"key":"ref7","doi-asserted-by":"crossref","first-page":"k1479","DOI":"10.1136\/bmj.k1479","article-title":"Biases in electronic health record data due to processes within the healthcare system: retrospective observational study","volume":"361","author":"D Agniel","year":"2018","journal-title":"BMJ"},{"issue":"04","key":"ref8","doi-asserted-by":"crossref","first-page":"1193","DOI":"10.1109\/JBHI.2015.2450362","article-title":"Big data for health","volume":"19","author":"J Andreu-Perez","year":"2015","journal-title":"IEEE J Biomed Health Inform"},{"issue":"08","key":"ref9","doi-asserted-by":"crossref","first-page":"e0220369","DOI":"10.1371\/journal.pone.0220369","article-title":"Temporal variability analysis reveals biases in electronic health records due to hospital process reengineering interventions over seven years","volume":"14","author":"F J P\u00e9rez-Benito","year":"2019","journal-title":"PLoS One"},{"issue":"02","key":"ref10","doi-asserted-by":"crossref","first-page":"e034396","DOI":"10.1136\/bmjopen-2019-034396","article-title":"Data-driven discovery of changes in clinical code usage over time: a case-study on changes in cardiovascular disease recording in two English electronic health records databases (2001-2015)","volume":"10","author":"P Rockenschaub","year":"2020","journal-title":"BMJ Open"},{"issue":"08","key":"ref11","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1176\/appi.ps.201600068","article-title":"Facilitating action for suicide prevention by learning health care systems","volume":"67","author":"R C Rossom","year":"2016","journal-title":"Psychiatr Serv"},{"issue":"01","key":"ref12","first-page":"1049","article-title":"The HMO research network virtual data warehouse: a public data model to support collaboration","volume":"2","author":"T R Ross","year":"2014","journal-title":"EGEMS (Wash DC)"},{"issue":"11","key":"ref13","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.1016\/0895-4356(94)90129-5","article-title":"Validation of a combined comorbidity index","volume":"47","author":"M Charlson","year":"1994","journal-title":"J Clin Epidemiol"},{"issue":"01","key":"ref14","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"R Tibshirani","year":"1996","journal-title":"J R Stat Soc Series B Stat Methodol"},{"issue":"01","key":"ref15","doi-asserted-by":"crossref","first-page":"121","DOI":"10.2307\/2531248","article-title":"Longitudinal data analysis for discrete and continuous outcomes","volume":"42","author":"S L Zeger","year":"1986","journal-title":"Biometrics"},{"key":"ref16","volume-title":"R: A Language and Environment for Statistical Computing","author":"R Core Team","year":"2019"},{"issue":"20","key":"ref17","doi-asserted-by":"crossref","first-page":"3940","DOI":"10.1093\/bioinformatics\/bti623","article-title":"ROCR: visualizing classifier performance in R","volume":"21","author":"T Sing","year":"2005","journal-title":"Bioinformatics"},{"issue":"12","key":"ref18","doi-asserted-by":"crossref","first-page":"1645","DOI":"10.1093\/jamia\/ocz145","article-title":"Prognostic models will be victims of their own success, unless\u2026","volume":"26","author":"M C Lenert","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"ref19","doi-asserted-by":"crossref","first-page":"1085","DOI":"10.1016\/j.jclinepi.2008.04.008","article-title":"Validation, updating and impact of clinical prediction rules: a review","volume":"61","author":"D B Toll","year":"2008","journal-title":"J Clin Epidemiol"},{"issue":"06","key":"ref20","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1093\/jamia\/ocx030","article-title":"Calibration drift in regression and machine learning models for acute kidney injury","volume":"24","author":"S E Davis","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"ref21","doi-asserted-by":"crossref","first-page":"1458","DOI":"10.1093\/jamia\/ocz136","article-title":"What health records data are required for accurate prediction of suicidal behavior?","volume":"26","author":"G E Simon","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"ref22","first-page":"191","article-title":"A review of challenges and opportunities in machine learning for health","volume":"2020","author":"M Ghassemi","year":"2020","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"issue":"01","key":"ref23","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1038\/s41398-020-0780-3","article-title":"Deep learning in mental health outcome research: a scoping review","volume":"10","author":"C Su","year":"2020","journal-title":"Transl Psychiatry"},{"issue":"01","key":"ref24","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review","volume":"24","author":"B A Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"ref25","doi-asserted-by":"crossref","first-page":"1651","DOI":"10.1093\/jamia\/ocz130","article-title":"Predictive analytics in health care: how can we know it works?","volume":"26","author":"B Van Calster","year":"2019","journal-title":"J Am Med Inform Assoc"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0041-1733908.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,6]],"date-time":"2024-09-06T20:58:51Z","timestamp":1725656331000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0041-1733908"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8]]},"references-count":25,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2021,8,4]]},"published-print":{"date-parts":[[2021,8]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0041-1733908","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8]]}}}