{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T15:44:30Z","timestamp":1772120670822,"version":"3.50.1"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2019,9,16]],"date-time":"2019-09-16T00:00:00Z","timestamp":1568592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>The study sought to evaluate how availability of different types of health records data affect the accuracy of machine learning models predicting suicidal behavior.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Records from 7 large health systems identified 19 061 056 outpatient visits to mental health specialty or general medical providers between 2009 and 2015. Machine learning models (logistic regression with penalized LASSO [least absolute shrinkage and selection operator] variable selection) were developed to predict suicide death (n\u2009=\u20091240) or probable suicide attempt (n\u2009=\u200924 133) in the following 90 days. Base models were used only historical insurance claims data and were then augmented with data regarding sociodemographic characteristics (race, ethnicity, and neighborhood characteristics), past patient-reported outcome questionnaires from electronic health records, and data (diagnoses and questionnaires) recorded during the visit.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>For prediction of any attempt following mental health specialty visits, a model limited to historical insurance claims data performed approximately as well (C-statistic 0.843) as a model using all available data (C-statistic 0.850). For prediction of suicide attempt following a general medical visit, addition of data recorded during the visit yielded a meaningful improvement over a model using all data up to the prior day (C-statistic 0.853 vs 0.838).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Results may not generalize to setting with less comprehensive data or different patterns of care. Even the poorest-performing models were superior to brief self-report questionnaires or traditional clinical assessment.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Implementation of suicide risk prediction models in mental health specialty settings may be less technically demanding than expected. In general medical settings, however, delivery of optimal risk predictions at the point of care may require more sophisticated informatics capability.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocz136","type":"journal-article","created":{"date-parts":[[2019,7,24]],"date-time":"2019-07-24T11:25:23Z","timestamp":1563967523000},"page":"1458-1465","source":"Crossref","is-referenced-by-count":35,"title":["What health records data are required for accurate prediction of suicidal behavior?"],"prefix":"10.1093","volume":"26","author":[{"given":"Gregory E","family":"Simon","sequence":"first","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA"}]},{"given":"Susan M","family":"Shortreed","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA"}]},{"given":"Eric","family":"Johnson","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA"}]},{"given":"Rebecca C","family":"Rossom","sequence":"additional","affiliation":[{"name":"HealthPartners Institute, Minneapolis, Minnesota, USA"}]},{"given":"Frances L","family":"Lynch","sequence":"additional","affiliation":[{"name":"Center for Health Research, Kaiser Permanente Northwest, Portland, Oregon, USA"}]},{"given":"Rebecca","family":"Ziebell","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA"}]},{"given":"and Robert B","family":"Penfold","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington, USA"}]}],"member":"286","published-online":{"date-parts":[[2019,9,16]]},"reference":[{"key":"2020110612451551800_ocz136-B1","volume-title":"Mortality in the United States, 2017","author":"Murphy","year":"2017"},{"key":"2020110612451551800_ocz136-B2","author":"WISQARS Nonfatal Injury Reports, 2000\u20132014","year":"2017"},{"issue":"6","key":"2020110612451551800_ocz136-B3","doi-asserted-by":"crossref","first-page":"870","DOI":"10.1007\/s11606-014-2767-3","article-title":"Health care contacts in the year before suicide death","volume":"29","author":"Ahmedani","year":"2014","journal-title":"J Gen Intern Med"},{"issue":"5","key":"2020110612451551800_ocz136-B4","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1097\/MLR.0000000000000335","article-title":"Racial\/ethnic differences in health care visits made before suicide attempt across the United States","volume":"53","author":"Ahmedani","year":"2015","journal-title":"Med Care"},{"key":"2020110612451551800_ocz136-B5","first-page":"1","article-title":"Detecting and treating suicidal ideation in all settings","volume":"56:","author":"Patient Safety Advisory Group","year":"2016","journal-title":"Sentinel Event Alerts"},{"issue":"2","key":"2020110612451551800_ocz136-B6","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1037\/bul0000084","article-title":"Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research","volume":"143","author":"Franklin","year":"2017","journal-title":"Psychol Bull"},{"issue":"2","key":"2020110612451551800_ocz136-B7","doi-asserted-by":"crossref","first-page":"221","DOI":"10.4088\/JCP.15m09776","article-title":"Risk of suicide attempt and suicide death following completion of the patient health questionnaire depression module in community practice","volume":"77","author":"Simon","year":"2016","journal-title":"J Clin Psychiatry"},{"issue":"5","key":"2020110612451551800_ocz136-B8","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1176\/appi.ps.201500149","article-title":"Does suicidal ideation as measured by the PHQ-9 predict suicide among VA patients?","volume":"67","author":"Louzon","year":"2016","journal-title":"Psychiatr Serv"},{"issue":"4","key":"2020110612451551800_ocz136-B9","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1038\/mp.2016.110","article-title":"Predicting suicides after outpatient mental health visits in the Army Study to Assess Risk and Resilience in Service members (Army STARRS)","volume":"22","author":"Kessler","year":"2017","journal-title":"Mol Psychiatry"},{"issue":"9","key":"2020110612451551800_ocz136-B10","doi-asserted-by":"crossref","first-page":"1935","DOI":"10.2105\/AJPH.2015.302737","article-title":"Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs","volume":"105","author":"McCarthy","year":"2015","journal-title":"Am J Public Health"},{"issue":"3","key":"2020110612451551800_ocz136-B11","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1177\/2167702617691560","article-title":"Predicting risk of suicide attempts over time through machine learning","volume":"5","author":"Walsh","year":"2017","journal-title":"Clin Psychol Sci"},{"issue":"10","key":"2020110612451551800_ocz136-B12","doi-asserted-by":"crossref","first-page":"1064","DOI":"10.1001\/jamapsychiatry.2016.2172","article-title":"Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing","volume":"73","author":"McCoy","year":"2016","journal-title":"JAMA Psychiatry"},{"issue":"1","key":"2020110612451551800_ocz136-B13","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1001\/jamapsychiatry.2014.1754","article-title":"Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and resilience in Service members (Army STARRS)","volume":"72","author":"Kessler","year":"2015","journal-title":"JAMA Psychiatry"},{"issue":"2","key":"2020110612451551800_ocz136-B14","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1176\/appi.ajp.2016.16010077","article-title":"Predicting suicidal behavior from longitudinal electronic health records","volume":"174","author":"Barak-Corren","year":"2017","journal-title":"Am J Psychiatry"},{"issue":"3","key":"2020110612451551800_ocz136-B15","doi-asserted-by":"crossref","first-page":"e1575","DOI":"10.1002\/mpr.1575","article-title":"Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration","volume":"26","author":"Kessler","year":"2017","journal-title":"Int J Methods Psychiatr Res"},{"issue":"10","key":"2020110612451551800_ocz136-B16","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1176\/appi.ajp.2018.17101167","article-title":"Predicting Suicide attempts and suicide deaths following outpatient visits using electronic health records","volume":"175","author":"Simon","year":"2018","journal-title":"Am J Psychiatry"},{"issue":"4","key":"2020110612451551800_ocz136-B17","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1016\/j.genhosppsych.2010.03.006","article-title":"The patient health questionnaire somatic, anxiety, and depressive symptom scales: a systematic review","volume":"32","author":"Kroenke","year":"2010","journal-title":"Gen Hosp Psychiatry"},{"issue":"1","key":"2020110612451551800_ocz136-B18","first-page":"2","article-title":"The HMO Research network virtual data warehouse: a public data model to support collaboration","volume":"2","author":"Ross","year":"2014","journal-title":"EGEMS (Wash DC)"},{"issue":"11","key":"2020110612451551800_ocz136-B19","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.1016\/0895-4356(94)90129-5","article-title":"Validation of a combined comorbidity index","volume":"47","author":"Charlson","year":"1994","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2020110612451551800_ocz136-B20","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1007\/s00127-015-1119-x","article-title":"The accuracy of suicide statistics: are true suicide deaths misclassified?","volume":"51","author":"Bakst","year":"2016","journal-title":"Soc Psychiatry Psychiatr Epidemiol"},{"issue":"3","key":"2020110612451551800_ocz136-B21","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1111\/sltb.12280","article-title":"An examination of potential misclassification of army suicides: results from the Army Study to Assess Risk and Resilience in Service members","volume":"47","author":"Cox","year":"2017","journal-title":"Suicide Life Threat Behav"},{"key":"2020110612451551800_ocz136-B22","first-page":"267","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J R Stat Soc Series B Stat Methodol"},{"key":"2020110612451551800_ocz136-B23","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The Elements of Statistical Learning","author":"Hastie","year":"2009","edition":"2nd ed."},{"issue":"430","key":"2020110612451551800_ocz136-B24","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1080\/01621459.1995.10476572","article-title":"Bayes Factors","volume":"90","author":"Kass","year":"1995","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"2020110612451551800_ocz136-B25","doi-asserted-by":"crossref","first-page":"13.","DOI":"10.1093\/biomet\/73.1.13","article-title":"Longitudinal data analysis using generalized linear models","volume":"73","author":"Liang","year":"1986","journal-title":"Biometrika"},{"key":"2020110612451551800_ocz136-B26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v033.i01","article-title":"Regularization paths for generalized linear models via coordinate descent","volume":"33","author":"Friedman","year":"2010","journal-title":"J Stat Softw"},{"key":"2020110612451551800_ocz136-B27","author":"Weston"},{"key":"2020110612451551800_ocz136-B28","volume-title":"Signal Detection Theory and ROC Analysis","author":"Egan","year":"1975"},{"issue":"1","key":"2020110612451551800_ocz136-B29","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","article-title":"The meaning and use of the area under a receiver operating characteristic (ROC) curve","volume":"143","author":"Hanley","year":"1982","journal-title":"Radiology"},{"issue":"7","key":"2020110612451551800_ocz136-B30","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1016\/S0031-3203(96)00142-2","article-title":"The use of the area under the ROC curve in the evaluation of machine learning algorithms","volume":"30","author":"Bradley","year":"1997","journal-title":"Pattern Recognit"},{"issue":"10","key":"2020110612451551800_ocz136-B31","doi-asserted-by":"crossref","first-page":"1051","DOI":"10.1176\/appi.ps.201400283","article-title":"Evaluation of veterans' suicide risk with the use of linguistic detection methods","volume":"66","author":"Leonard Westgate","year":"2015","journal-title":"Psychiatr Serv"},{"issue":"1","key":"2020110612451551800_ocz136-B32","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1176\/appi.ps.201800242","article-title":"Integrating predictive modeling into mental health care: an example in suicide prevention","volume":"70","author":"Reger","year":"2019","journal-title":"Psychiatr Serv"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1458\/34151748\/ocz136.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/12\/1458\/34151748\/ocz136.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:28:12Z","timestamp":1604687292000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/12\/1458\/5570544"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,16]]},"references-count":32,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2019,9,16]]},"published-print":{"date-parts":[[2019,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz136","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,12]]},"published":{"date-parts":[[2019,9,16]]}}}