{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:47:16Z","timestamp":1792788436498,"version":"4.5.0"},"reference-count":44,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T00:00:00Z","timestamp":1604620800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010269","name":"The Wellcome Trust","doi-asserted-by":"crossref","award":["202836\/Z\/16\/Z"],"award-info":[{"award-number":["202836\/Z\/16\/Z"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"crossref"}]},{"name":"The European Union\u2019s Seventh Framework Programme for research, technological development and demonstration","award":["602805"],"award-info":[{"award-number":["602805"]}]},{"name":"The European Union\u2019s Horizon 2020 research and innovation programme","award":["667302"],"award-info":[{"award-number":["667302"]}]},{"name":"The European Union\u2019s Seventh Framework Programme for research, technological development and demonstration","award":["602805"],"award-info":[{"award-number":["602805"]}]},{"name":"The European Union\u2019s Horizon 2020 research and innovation programme","award":["667302"],"award-info":[{"award-number":["667302"]}]},{"name":"The European Union\u2019s Horizon 2020 research and innovation programme","award":["728018"],"award-info":[{"award-number":["728018"]}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"crossref","award":["5R01MH101519"],"award-info":[{"award-number":["5R01MH101519"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"crossref","award":["U01 MH109536-01"],"award-info":[{"award-number":["U01 MH109536-01"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosmedicine.org"],"crossmark-restriction":false},"short-container-title":["PLoS Med"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Suicide is a major public health concern globally. Accurately predicting suicidal behavior remains challenging. This study aimed to use machine learning approaches to examine the potential of the Swedish national registry data for prediction of suicidal behavior.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods and findings<\/jats:title>\n                    <jats:p>\n                      The study sample consisted of 541,300 inpatient and outpatient visits by 126,205 Sweden-born patients (54% female and 46% male) aged 18 to 39 (mean age at the visit: 27.3) years to psychiatric specialty care in Sweden between January 1, 2011 and December 31, 2012. The most common psychiatric diagnoses at the visit were anxiety disorders (20.0%), major depressive disorder (16.9%), and substance use disorders (13.6%). A total of 425 candidate predictors covering demographic characteristics, socioeconomic status (SES), electronic medical records, criminality, as well as family history of disease and crime were extracted from the Swedish registry data. The sample was randomly split into an 80% training set containing 433,024 visits and a 20% test set containing 108,276 visits. Models were trained separately for suicide attempt\/death within 90 and 30 days following a visit using multiple machine learning algorithms. Model discrimination and calibration were both evaluated. Among all eligible visits, 3.5% (18,682) were followed by a suicide attempt\/death within 90 days and 1.7% (9,099) within 30 days. The final models were based on ensemble learning that combined predictions from elastic net penalized logistic regression, random forest, gradient boosting, and a neural network. The area under the receiver operating characteristic (ROC) curves (AUCs) on the test set were 0.88 (95% confidence interval [CI] = 0.87\u20130.89) and 0.89 (95% CI = 0.88\u20130.90) for the outcome within 90 days and 30 days, respectively, both being significantly better than chance (i.e., AUC = 0.50) (\n                      <jats:italic>p<\/jats:italic>\n                      &lt; 0.01). Sensitivity, specificity, and predictive values were reported at different risk thresholds. A limitation of our study is that our models have not yet been externally validated, and thus, the generalizability of the models to other populations remains unknown.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>By combining the ensemble method of multiple machine learning algorithms and high-quality data solely from the Swedish registers, we developed prognostic models to predict short-term suicide attempt\/death with good discrimination and calibration. Whether novel predictors can improve predictive performance requires further investigation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pmed.1003416","type":"journal-article","created":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T13:22:42Z","timestamp":1604668962000},"page":"e1003416","update-policy":"https:\/\/doi.org\/10.1371\/journal.pmed.corrections_policy","source":"Crossref","is-referenced-by-count":77,"title":["Predicting suicide attempt or suicide death following a visit to psychiatric specialty care: A machine learning study using Swedish national registry data"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4602-4388","authenticated-orcid":true,"given":"Qi","family":"Chen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2104-0963","authenticated-orcid":true,"given":"Yanli","family":"Zhang-James","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric J.","family":"Barnett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Lichtenstein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jussi","family":"Jokinen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5248-7434","authenticated-orcid":true,"given":"Brian M.","family":"D\u2019Onofrio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9217-3982","authenticated-orcid":true,"given":"Stephen V.","family":"Faraone","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henrik","family":"Larsson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5383-5365","authenticated-orcid":true,"given":"Seena","family":"Fazel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2020,11,6]]},"reference":[{"issue":"2","key":"pmed.1003416.ref001","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1037\/bul0000084","article-title":"Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research","volume":"143","author":"JC Franklin","year":"2017","journal-title":"Psychol Bull"},{"issue":"7","key":"pmed.1003416.ref002","doi-asserted-by":"crossref","first-page":"e0180292","DOI":"10.1371\/journal.pone.0180292","article-title":"Instruments for the assessment of suicide risk: A systematic review evaluating the certainty of the evidence","volume":"12","author":"B Runeson","year":"2017","journal-title":"PLoS One"},{"issue":"6","key":"pmed.1003416.ref003","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1192\/bjp.bp.116.182717","article-title":"Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales","volume":"210","author":"G Carter","year":"2017","journal-title":"Br J Psychiatry"},{"key":"pmed.1003416.ref004","article-title":"Machine learning in suicide science: Applications and ethics.","author":"KP Linthicum","year":"2019","journal-title":"Behav Sci Law"},{"issue":"3","key":"pmed.1003416.ref005","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1136\/ebmental-2019-300102","article-title":"How accurate are suicide risk prediction models? Asking the right questions for clinical practice","volume":"22","author":"D Whiting","year":"2019","journal-title":"Evid Based Ment Health"},{"key":"pmed.1003416.ref006","article-title":"Prediction Models for Suicide Attempts and Deaths: A Systematic Review and Simulation","author":"BE Belsher","year":"2019","journal-title":"JAMA Psychiat"},{"issue":"1","key":"pmed.1003416.ref007","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1038\/s41398-019-0428-3","article-title":"The prediction of suicide in severe mental illness: development and validation of a clinical prediction rule (OxMIS)","volume":"9","author":"S Fazel","year":"2019","journal-title":"Transl Psychiatry."},{"issue":"11","key":"pmed.1003416.ref008","doi-asserted-by":"crossref","first-page":"659","DOI":"10.1007\/s10654-009-9350-y","article-title":"The Swedish personal identity number: possibilities and pitfalls in healthcare and medical research","volume":"24","author":"JF Ludvigsson","year":"2009","journal-title":"Eur J Epidemiol"},{"key":"pmed.1003416.ref009","author":"National Board of Health and Welfare","year":"2014","journal-title":"Yearbook of the Swedish Medical Birth Registry"},{"issue":"2","key":"pmed.1003416.ref010","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/s10654-016-0117-y","article-title":"Registers of the Swedish total population and their use in medical research","volume":"31","author":"JF Ludvigsson","year":"2016","journal-title":"Eur J Epidemiol"},{"key":"pmed.1003416.ref011","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/978-1-59745-423-0_10","article-title":"The Swedish Multi-generation Register","volume":"675","author":"A Ekbom","year":"2011","journal-title":"Methods Mol Biol"},{"key":"pmed.1003416.ref012","article-title":"The longitudinal integrated database for health insurance and labour market studies (LISA) and its use in medical research","author":"JF Ludvigsson","year":"2019","journal-title":"Eur J Epidemiol"},{"key":"pmed.1003416.ref013","doi-asserted-by":"crossref","first-page":"450","DOI":"10.1186\/1471-2458-11-450","article-title":"External review and validation of the Swedish national inpatient register","volume":"11","author":"JF Ludvigsson","year":"2011","journal-title":"BMC Public Health"},{"issue":"7","key":"pmed.1003416.ref014","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1002\/pds.1294","article-title":"The new Swedish Prescribed Drug Register\u2014opportunities for pharmacoepidemiological research and experience from the first six months","volume":"16","author":"B Wettermark","year":"2007","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"10","key":"pmed.1003416.ref015","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1016\/S2215-0366(15)00234-5","article-title":"Psychiatric disorders and violent reoffending: a national cohort study of convicted prisoners in Sweden.","volume":"2","author":"Z Chang","year":"2015","journal-title":"Lancet Psychiatry"},{"key":"pmed.1003416.ref016","article-title":"Predicting Suicide Attempts and Suicide Deaths Following Outpatient Visits Using Electronic Health Records","author":"GE Simon","year":"2018","journal-title":"Am J Psychiatry"},{"issue":"3","key":"pmed.1003416.ref017","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1177\/2167702617691560","article-title":"Predicting Risk of Suicide Attempts Over Time Through Machine Learning","volume":"5","author":"CG Walsh","year":"2017","journal-title":"Clin Psychol Sci"},{"issue":"2","key":"pmed.1003416.ref018","doi-asserted-by":"crossref","first-page":"e003497","DOI":"10.1136\/bmjopen-2013-003497","article-title":"Criminality and suicide: a longitudinal Swedish cohort study","volume":"4","author":"M Stenbacka","year":"2014","journal-title":"BMJ Open"},{"issue":"3","key":"pmed.1003416.ref019","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1111\/acps.12458","article-title":"The role of inflammation in suicidal behaviour","volume":"132","author":"L Brundin","year":"2015","journal-title":"Acta Psychiatr Scand"},{"issue":"2","key":"pmed.1003416.ref020","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"H Zou","year":"2005","journal-title":"J R Stat Soc Series B Stat Methodol"},{"issue":"1","key":"pmed.1003416.ref021","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests.","volume":"45","author":"L Breiman","year":"2001","journal-title":"Mach Learn"},{"key":"pmed.1003416.ref022","doi-asserted-by":"crossref","unstructured":"Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; San Francisco, California, USA. 2939785: ACM; 2016. p. 785\u201394.","DOI":"10.1145\/2939672.2939785"},{"key":"pmed.1003416.ref023","first-page":"842","volume-title":"Neural Networks","author":"S Haykin","year":"1998"},{"key":"pmed.1003416.ref024","first-page":"1","volume-title":"Proceedings of the First International Workshop on Multiple Classifier Systems. 743935","author":"TG Dietterich","year":"2000"},{"issue":"3","key":"pmed.1003416.ref025","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"ER DeLong","year":"1988","journal-title":"Biometrics"},{"key":"pmed.1003416.ref026","doi-asserted-by":"crossref","first-page":"307","DOI":"10.3389\/fpubh.2017.00307","article-title":"Sensitivity, Specificity, and Predictive Values: Foundations, Pliabilities, and Pitfalls in Research and Practice","volume":"5","author":"R Trevethan","year":"2017","journal-title":"Front Public Health."},{"key":"pmed.1003416.ref027","doi-asserted-by":"crossref","first-page":"h4978","DOI":"10.1136\/bmj.h4978","article-title":"Suicide risk assessment and intervention in people with mental illness","volume":"351","author":"JM Bolton","year":"2015","journal-title":"BMJ"},{"key":"pmed.1003416.ref028","doi-asserted-by":"crossref","DOI":"10.1145\/1102351.1102430","volume-title":"Predicting Good Probabilities With Supervised Learning","author":"A Niculescu-Mizil","year":"2005"},{"key":"pmed.1003416.ref029","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"78","author":"GW Brier","year":"1950","journal-title":"Mon Weather Rev"},{"key":"pmed.1003416.ref030","first-page":"454","volume-title":"Python Machine Learning","author":"S Raschka","year":"2015"},{"issue":"1","key":"pmed.1003416.ref031","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1148\/radiology.143.1.7063747","article-title":"The meaning and use of the area under a receiver operating characteristic (ROC) curve","volume":"143","author":"JA Hanley","year":"1982","journal-title":"Radiology"},{"issue":"2","key":"pmed.1003416.ref032","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1037\/abn0000317","article-title":"Risk factors for the transition from suicide ideation to suicide attempt: Results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).","volume":"127","author":"MK Nock","year":"2018","journal-title":"J Abnorm Psychol"},{"key":"pmed.1003416.ref033","doi-asserted-by":"crossref","unstructured":"Rao RB, Glenn F, Romer R. On the Dangers of Cross-Validation. An Experimental Evaluation. In Proceedings of the 2008 SIAM International Conference on Data Mining; 2008. pp. 588\u2013596.","DOI":"10.1137\/1.9781611972788.54"},{"key":"pmed.1003416.ref034","first-page":"546671","article-title":"Machine Learning Classification of Attention-Deficit\/Hyperactivity Disorder Using Structural MRI Data","author":"Y Zhang-James","year":"2019"},{"key":"pmed.1003416.ref035","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1016\/j.jad.2019.03.044","article-title":"Short-term prediction of suicidal thoughts and behaviors in adolescents: Can recent developments in technology and computational science provide a breakthrough?","volume":"250","author":"NB Allen","year":"2019","journal-title":"J Affect Disord"},{"issue":"7553","key":"pmed.1003416.ref036","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"Y LeCun","year":"2015","journal-title":"Nature"},{"key":"pmed.1003416.ref037","first-page":"775","volume-title":"Deep learning","author":"I Goodfellow","year":"2016"},{"issue":"8","key":"pmed.1003416.ref038","doi-asserted-by":"crossref","first-page":"568","DOI":"10.1016\/S1047-2797(97)00126-9","article-title":"Risk of suicide attempts after benzodiazepine and\/or antidepressant use","volume":"7","author":"CI Neutel","year":"1997","journal-title":"Ann Epidemiol"},{"issue":"2","key":"pmed.1003416.ref039","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1002\/wps.20128","article-title":"Risks of all-cause and suicide mortality in mental disorders: a meta-review","volume":"13","author":"E Chesney","year":"2014","journal-title":"World Psychiatry."},{"key":"pmed.1003416.ref040","article-title":"Machine Learning for Suicide Research-Can It Improve Risk Factor Identification?","author":"S Fazel","year":"2019","journal-title":"JAMA Psychiat"},{"key":"pmed.1003416.ref041","article-title":"Prediction of Sex-Specific Suicide Risk Using Machine Learning and Single-Payer Health Care Registry Data From Denmark","author":"JL Gradus","year":"2019","journal-title":"JAMA Psychiat"},{"key":"pmed.1003416.ref042","doi-asserted-by":"crossref","first-page":"c3222","DOI":"10.1136\/bmj.c3222","article-title":"Method of attempted suicide as predictor of subsequent successful suicide: national long term cohort study","volume":"341","author":"B Runeson","year":"2010","journal-title":"BMJ"},{"issue":"9","key":"pmed.1003416.ref043","doi-asserted-by":"crossref","DOI":"10.3390\/ijerph15092028","article-title":"Suicide Risk and Mental Disorders.","volume":"15","author":"L Bradvik","year":"2018","journal-title":"Int J Environ Res Public Health"},{"key":"pmed.1003416.ref044","first-page":"2018","article-title":"All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously","author":"A Fisher"}],"container-title":["PLOS Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pmed.1003416","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T13:23:44Z","timestamp":1604669024000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pmed.1003416"}},"subtitle":[],"editor":[{"given":"Alexander C.","family":"Tsai","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2020,11,6]]},"references-count":44,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,11,6]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pmed.1003416","relation":{},"ISSN":["1549-1676"],"issn-type":[{"value":"1549-1676","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,6]]}}}