{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T21:19:53Z","timestamp":1787779193019,"version":"build-2784847793"},"reference-count":52,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Suicidal behavior is a major public health problem worldwide. The exact etiology remains unclear, representing a complex problem involving multiple factors. Evidence indicates that around 50% to 80% of people who die by suicide have had contact with the health care system in the year prior to their death.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We present the IDICIUS project, whose objective is to develop a clinical decision support system that functions as an early warning system and applies AI to prevent suicide risk using anonymized electronic health record data.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This study shows the first 2 stages of the IDICIUS project, where real-world data from 4 public sources were integrated, curated, standardized, and anonymized in phase 1, and analysis and modeling of suicide risk were performed in phase 2. This retrospective population-based study included 41,557 adult patients receiving mental health care at a large hospital. We evaluated the performance of machine learning classifiers with increasing complexity: logistic regression, elastic net, decision trees and random forests, bagging and boosting ensemble methods (GradientBoosting, XGBoost [extreme gradient boosting], CatBoost, RandomForest, and AdaBoost [adaptive boosting]), support vector machines, and deep neural networks. To address class imbalance, we tested several balancing techniques, with random undersampling providing the best results.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Phase 1 yielded a useful database of 32,661 patients receiving mental health care and 112 features. Of those, 2764 patients exhibited suicidal behavior (target), while 29,897 did not (control). The undersampling reduced the class distribution to 4422 vs 2211 (control vs target). The most prevalent features in the target class (N=2764) were psychiatric conditions, anxiety episodes (n=2269, 82.1%) and depressive episodes (n=1682, 60.9%), along with demographic and behavioral factors: female sex (n=1672, 60.5%), alcohol consumption (n=770, 27.9%), and conduct disorder (n=709, 25.7%). The largest patient subgroups were females with combined depression and anxiety (n=300, 10.9%), females with anxiety disorders only (n=182, 6.6%), males with anxiety disorders only (n=120, 4.3%), and males with combined depression and anxiety (n=106, 3.8%). In phase 2, ensemble methods, especially GradientBoosting and XGBoost, achieved the best performance, with receiver operating characteristic-area under the curve scores around 0.95. While these models showed moderate precision in identifying true positives (0.51\u20100.58), they demonstrated high sensitivity in detecting at-risk patients, with recall scores of 0.85 and 0.84, respectively. Both models achieved a good balance between precision and recall (\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores: 0.68 and 0.67). The strongest predictors were corticosteroid use, psychiatric medications (olanzapine and lorazepam), female sex, and depression.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n We integrated multiple electronic health record data sources and applied AI to develop, model, and optimize ensemble algorithms for suicide prevention, maximizing effectiveness, efficiency, and generalizability. These AI-generated algorithms, based on readily available and well-structured data, may advance the identification of at-risk patients and function as early warning systems, thereby increasing opportunities for timely preventive interventions.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/85085","type":"journal-article","created":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T18:50:07Z","timestamp":1781549407000},"page":"e85085-e85085","source":"Crossref","is-referenced-by-count":0,"title":["AI Models for Suicide Risk Prediction in Adult Patients Receiving Mental Health Care Using Real-World Data: Retrospective Population-Based Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8043-8187","authenticated-orcid":false,"given":"Cleof\u00e9","family":"Pe\u00f1a-G\u00f3mez","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9290-6616","authenticated-orcid":false,"given":"Marc","family":"Fradera","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4335-6851","authenticated-orcid":false,"given":"Xavier","family":"S\u00e1nchez Corrales","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1094-0023","authenticated-orcid":false,"given":"Marc","family":"Caravaca-Rodriguez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6711-4956","authenticated-orcid":false,"given":"Juan-Francisco","family":"Mart\u00ednez-Cerd\u00e1","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0256-2942","authenticated-orcid":false,"given":"Joan Albert","family":"Escofet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7524-4590","authenticated-orcid":false,"given":"David","family":"Roche","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5804-3597","authenticated-orcid":false,"given":"Eneko","family":"Barber\u00eda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3274-6048","authenticated-orcid":false,"given":"Caridad","family":"Pontes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7082-4695","authenticated-orcid":false,"given":"Jes\u00fas","family":"Giraldo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3323-6568","authenticated-orcid":false,"given":"Diego","family":"Palao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,26]]},"reference":[{"key":"R1","unstructured":"LIVE LIFE: an implementation guide for suicide prevention in countries. World Health Organization; 2021. URL: https:\/\/www.who.int\/publications\/i\/item\/9789240026629 [Accessed 01-05-2025]"},{"key":"R2","unstructured":"Preventing suicide: a global imperative. World Health Organization; 2014. URL: https:\/\/iris.who.int\/server\/api\/core\/bitstreams\/5bd12922-b362-458f-8025-f613da4922e7\/content [Accessed 23-07-2026]"},{"key":"R3","unstructured":"Suicide worldwide in 2021: global health estimates. World Health Organization; 2025. URL: https:\/\/iris.who.int\/server\/api\/core\/bitstreams\/769d0a45-b50a-4b17-ba40-259bef44d9dd\/content [Accessed 23-07-2026]"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1037\/bul0000084","article-title":"Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research","volume":"143","author":"Franklin","journal-title":"Psychol Bull"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2020.05.071","article-title":"Contacts with health services during the year prior to suicide death and prevalent conditions a nationwide study","volume":"274","author":"Laanani","journal-title":"J Affect Disord"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1177\/1403494817746274","article-title":"Contact with primary and mental health care prior to suicide: a systematic review of the literature from 2000 to 2017","volume":"47","author":"Stene-Larsen","journal-title":"Scand J Public Health"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"1395","DOI":"10.1080\/13811118.2021.1911894","article-title":"Can routine primary care records help in detecting suicide risk? A population-based case-control study in Barcelona","volume":"26","author":"Fradera","journal-title":"Arch Suicide Res"},{"issue":"7","key":"R8","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.eurpsy.2009.12.004","article-title":"Suicidal ideation, self-harm and attempted suicide: results from the British psychiatric morbidity survey 2000","volume":"25","author":"Bebbington","journal-title":"Eur Psychiatry"},{"issue":"2","key":"R9","doi-asserted-by":"publisher","DOI":"10.4081\/mi.2012.e15","article-title":"The hypothesis of a continuum in suicidality: a discussion on its validity and practical implications","volume":"4","author":"Sveticic","journal-title":"Ment Illn"},{"issue":"4","key":"R10","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1192\/bjp.bp.115.170050","article-title":"Predicting suicide following self-harm: systematic review of risk factors and risk scales","volume":"209","author":"Chan","journal-title":"Br J Psychiatry"},{"issue":"6","key":"R11","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0156322","article-title":"Meta-analysis of longitudinal cohort studies of suicide risk assessment among psychiatric patients: heterogeneity in results and lack of improvement over time","volume":"11","author":"Large","journal-title":"PLoS One"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1186\/s12888-017-1458-0","article-title":"Effectiveness of online and mobile telephone applications (\u2018apps\u2019) for the self-management of suicidal ideation and self-harm: a systematic review and meta-analysis","volume":"17","author":"Witt","journal-title":"BMC Psychiatry"},{"issue":"11","key":"R13","doi-asserted-by":"publisher","first-page":"1801","DOI":"10.1017\/S1041610217001430","article-title":"A systematic review of interventions to prevent suicidal behaviors and reduce suicidal ideation in older people","volume":"29","author":"Okolie","journal-title":"Int Psychogeriatr"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2016.12.035","article-title":"The effectiveness of suicide prevention delivered by GPs: a systematic review and meta-analysis","volume":"210","author":"Milner","journal-title":"J Affect Disord"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1111\/sltb.12395","article-title":"Accuracy of clinician predictions of future self\u2010harm: a systematic review and meta\u2010analysis of predictive studies","volume":"49","author":"Woodford","journal-title":"Suicide Life Threat Behav"},{"issue":"5","key":"R16","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1176\/appi.ps.202000214","article-title":"Reconciling statistical and clinicians\u2019 predictions of suicide risk","volume":"72","author":"Simon","journal-title":"Psychiatr Serv"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1176\/appi.ajp.2016.16010077","article-title":"Predicting suicidal behavior from longitudinal electronic health records","volume":"174","author":"Barak-Corren","journal-title":"Am J Psychiatry"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1001\/jamapsychiatry.2014.1754","article-title":"Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS)","volume":"72","author":"Kessler","journal-title":"JAMA Psychiatry"},{"issue":"10","key":"R19","doi-asserted-by":"publisher","first-page":"951","DOI":"10.1176\/appi.ajp.2018.17101167","article-title":"Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records","volume":"175","author":"Simon","journal-title":"Am J Psychiatry"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0085733","article-title":"Predicting the risk of suicide by analyzing the text of clinical notes","volume":"9","author":"Poulin","journal-title":"PLoS One"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Ben-Ari A Hammond K . Text mining the EMR for modeling and predicting suicidal behavior among US veterans of the 1991 persian gulf war. Presented at: 2015 48th Hawaii International Conference on System Sciences (HICSS); Jan 5-8, 2015; Kauai, HI, USA. [doi: 10.1109\/HICSS.2015.382]","DOI":"10.1109\/HICSS.2015.382"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1038\/s41398-020-0780-3","article-title":"Deep learning in mental health outcome research: a scoping review","volume":"10","author":"Su","journal-title":"Transl Psychiatry"},{"issue":"S1","key":"R23","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1002\/pds.2335","article-title":"A systematic review of validated methods for identifying suicide or suicidal ideation using administrative or claims data","volume":"21","author":"Walkup","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"S1","key":"R24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/pds.3230","article-title":"The U.S. Food and Drug Administration\u2019s Mini\u2010sentinel Program","volume":"21","author":"Platt","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"McArdle JJ Ritschard G . Contemporary Issues in Exploratory Data Mining in the Behavioral Sciences. Routledge\/Taylor & Francis Group; 2014. [doi: 10.4324\/9780203403020]","DOI":"10.4324\/9780203403020"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1177\/2167702617691560","article-title":"Predicting risk of suicide attempts over time through machine learning","volume":"5","author":"Walsh","journal-title":"Clin Psychol Sci"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2018.11.073","article-title":"The use of machine learning in the study of suicidal and non-suicidal self-injurious thoughts and behaviors: a systematic review","volume":"245","author":"Burke","journal-title":"J Affect Disord"},{"issue":"3","key":"R28","doi-asserted-by":"publisher","DOI":"10.1002\/mpr.1575","article-title":"Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans health Administration","volume":"26","author":"Kessler","journal-title":"Int J Methods Psychiatr Res"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","DOI":"10.1038\/s41398-020-0684-2","article-title":"Development of an early-warning system for high-risk patients for suicide attempt using deep learning and electronic health records","volume":"10","author":"Zheng","journal-title":"Transl Psychiatry"},{"issue":"10","key":"R30","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.38986","article-title":"Early warning scores with and without artificial intelligence","volume":"7","author":"Edelson","journal-title":"JAMA Netw Open"},{"issue":"9","key":"R31","doi-asserted-by":"publisher","first-page":"1339","DOI":"10.1097\/CCM.0000000000005492","article-title":"The impact of a machine learning early warning score on hospital mortality: a multicenter clinical intervention trial","volume":"50","author":"Winslow","journal-title":"Crit Care Med"},{"issue":"20","key":"R32","doi-asserted-by":"publisher","first-page":"1951","DOI":"10.1056\/NEJMsa2001090","article-title":"Automated identification of adults at risk for in-hospital clinical deterioration","volume":"383","author":"Escobar","journal-title":"N Engl J Med"},{"key":"R33","unstructured":"Ley Org\u00e1nica 3\/2018, de 5 de diciembre, de Protecci\u00f3n de Datos Personales y garant\u00eda de los derechos digitales [Article in Spanish]. Bolet\u00edn Oficial del Estado (BOE). URL: https:\/\/www.boe.es\/eli\/es\/lo\/2018\/12\/05\/3\/con [Accessed 12-06-2018]"},{"key":"R34","unstructured":"Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation) (Text with EEA relevance) [Article in Spanish]. European Union; 2016. URL: https:\/\/eur-lex.europa.eu\/eli\/reg\/2016\/679\/oj [Accessed 05-08-2026]"},{"key":"R35","unstructured":"Lundberg SM Lee SI . A unified approach to interpreting model predictions. Presented at: NIPS\u201917: Proceedings of the 31st International Conference on Neural Information Processing Systems; Dec 4-9, 2017; Red Hook, NY, USA. [doi: 10.5555\/3295222.3295230]"},{"key":"R36","unstructured":"Florek P Zagda\u0144ski A . Benchmarking state-of-the-art gradient boosting algorithms for classification. arXiv. Preprint posted online on May 26, 2023. [doi: 10.48550\/arXiv.2305.17094]"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","DOI":"10.1136\/bmjment-2024-300990","article-title":"Suicide risk assessment tools and prediction models: new evidence, methodological innovations, outdated criticisms","volume":"27","author":"Seyedsalehi","journal-title":"BMJ Ment Health"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02524-0","article-title":"Role of machine learning algorithms in suicide risk prediction: a systematic review-meta analysis of clinical studies","volume":"24","author":"Ehtemam","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.1038\/s41398-024-02852-9","article-title":"Machine learning and the prediction of suicide in psychiatric populations: a systematic review","volume":"14","author":"Pigoni","journal-title":"Transl Psychiatry"},{"issue":"3","key":"R40","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/S2215-0366(21)00254-6","article-title":"Translating promise into practice: a review of machine learning in suicide research and prevention","volume":"9","author":"Kirtley","journal-title":"Lancet Psychiatry"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1192\/bjp.2025.128","article-title":"Oral glucocorticoids and risk of psychiatric and suicidal behaviour outcomes: population-based cohort study","volume":"229","author":"Lagerberg","journal-title":"Br J Psychiatry"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","DOI":"10.1186\/s12888-018-1702-2","article-title":"Insomnia as an independent predictor of suicide attempts: a nationwide population-based retrospective cohort study","volume":"18","author":"Lin","journal-title":"BMC Psychiatry"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.2196\/58933","article-title":"A 25-year retrospective of health IT infrastructure building: the example of the catalonia region","volume":"26","author":"Piera-Jim\u00e9nez","journal-title":"J Med Internet Res"},{"issue":"S1","key":"R44","doi-asserted-by":"publisher","DOI":"10.1192\/j.eurpsy.2025.335","article-title":"Machine learning model development to retrospectively predict suicide attempts in the Millenium Cohort Study sample","volume":"68","author":"Pe\u00f1a G\u00f3mez","journal-title":"Eur Psychiatr"},{"issue":"10","key":"R45","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.39269","article-title":"Performance of machine learning suicide risk models in an American Indian population","volume":"7","author":"Haroz","journal-title":"JAMA Netw Open"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-publhealth-051920-123206","article-title":"The recent rise of suicide mortality in the United States","volume":"43","author":"Mart\u00ednez-Al\u00e9s","journal-title":"Annu Rev Public Health"},{"issue":"4","key":"R47","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000023","article-title":"Generalizability challenges of mortality risk prediction models: a retrospective analysis on a multi-center database","volume":"1","author":"Singh","journal-title":"PLOS Digit Health"},{"issue":"10258","key":"R48","doi-asserted-by":"publisher","first-page":"1204","DOI":"10.1016\/S0140-6736(20)30925-9","article-title":"Global burden of 369 diseases and injuries in 204 countries and territories, 1990\u20132019: a systematic analysis for the Global Burden of Disease Study 2019","volume":"396","author":"Vos","journal-title":"The Lancet"},{"issue":"2","key":"R49","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1192\/bjp.2021.50","article-title":"Safety planning-type interventions for suicide prevention: meta-analysis","volume":"219","author":"Nuij","journal-title":"Br J Psychiatry"},{"key":"R50","doi-asserted-by":"publisher","DOI":"10.1016\/j.eclinm.2026.103824","article-title":"Effectiveness of brief interventions and contacts after suicide attempt: a systematic review and meta-analysis","volume":"93","author":"Homan","journal-title":"EClinicalMedicine"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2020.03.009","article-title":"The Catalonia Suicide Risk Code: a secondary prevention program for individuals at risk of suicide","volume":"268","author":"P\u00e9rez","journal-title":"J Affect Disord"},{"key":"R52","unstructured":"Ag\u00e8ncia de Qualitat i Avaluaci\u00f3 Sanit\u00e0ries de Catalunya (AQuAS) [Article in Catalan]. URL: https:\/\/aquas.gencat.cat\/ca\/inici [Accessed 23-07-2026]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T20:42:57Z","timestamp":1787776977000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e85085"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,26]]},"references-count":52,"URL":"https:\/\/doi.org\/10.2196\/85085","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,26]]},"article-number":"v5i2e85085"}}