{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T18:10:39Z","timestamp":1789150239499,"version":"build-2803163510"},"reference-count":36,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T00:00:00Z","timestamp":1789084800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["1177787"],"award-info":[{"award-number":["1177787"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["2008460"],"award-info":[{"award-number":["2008460"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003747","name":"Department of Health, State Government of Victoria","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003747","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100022580","name":"William Buckland Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100022580","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Future Generation Global"},{"DOI":"10.13039\/501100025520","name":"Medical Research Future Fund","doi-asserted-by":"crossref","award":["MRFCRI000188"],"award-info":[{"award-number":["MRFCRI000188"]}],"id":[{"id":"10.13039\/501100025520","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100025520","name":"Medical Research Future Fund","doi-asserted-by":"crossref","award":["MRFFRD000113"],"award-info":[{"award-number":["MRFFRD000113"]}],"id":[{"id":"10.13039\/501100025520","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Adequate self-harm surveillance is a key part of suicide prevention. Our previous research demonstrated that an artificial intelligence (AI)-based system could effectively detect self-harm in emergency department triage notes. However, the system was developed using data from a single hospital, raising concerns about its generalisability. Here, we aim to validate the system prospectively and externally to better understand its portability across hospitals. We leveraged emergency department data from two Australian hospitals, with free-text triage notes manually annotated for self-harm. The AI system was developed using data from a major metropolitan hospital in Melbourne, spanning 2012\u20132017. The system combined extensive text normalisation with a self-harm classification model utilising 1931 selected features. We prospectively validated the model on 329,655 triage notes from the same hospital over the following four years. For external validation, we used 316,877 triage notes from 2012 to 2021 from a regional hospital 150 km outside Melbourne. On the test set, the model achieved an area under the precision-recall curve (PR AUC) of 0.84 with a 95% confidence interval of [0.82, 0.86]. This performance remained stable at the development site with a PR AUC of 0.84, 95% CI [0.83, 0.85]. When applied in the regional context, the model\u2019s ability to distinguish self-harm cases declined, resulting in an overall PR AUC of 0.78, 95% CI [0.77, 0.79]. While the text normalisation component was equally effective across the datasets, our analysis revealed that in regional settings, self-harm presentations are more likely to involve medication ingestion. At the metropolitan hospital, the AI system for self-harm detection maintained its epidemiological utility. At the regional hospital, the text normalisation process was effective, but performance was unstable primarily due to linguistic domain shift and differences in self-harm presentations.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001667","type":"journal-article","created":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T17:42:48Z","timestamp":1789148568000},"page":"e0001667","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Temporal and cross-site validation of an AI system for self-harm detection"],"prefix":"10.1371","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1032-4650","authenticated-orcid":true,"given":"Vlada","family":"Rozova","sequence":"first","affiliation":[{"name":"Monash University","place":["Melbourne, Australia"],"department":["Department of Data Science and AI"]},{"name":"School of Computing and Information Systems, University of Melbourne","place":["Melbourne, Australia"]},{"name":"Centre for Digital Transformation of Health, University of Melbourne","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liuliu","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, University of Melbourne","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katrina","family":"Witt","sequence":"additional","affiliation":[{"name":"Centre for Youth Mental Health, The University of Melbourne","place":["Melbourne, Australia"]},{"name":"Orygen","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mike","family":"Conway","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, University of Melbourne","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jo","family":"Robinson","sequence":"additional","affiliation":[{"name":"Centre for Youth Mental Health, The University of Melbourne","place":["Melbourne, Australia"]},{"name":"Orygen","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8661-1544","authenticated-orcid":true,"given":"Karin","family":"Verspoor","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, University of Melbourne","place":["Melbourne, Australia"]},{"name":"School of Computing Technologies, RMIT University","place":["Melbourne, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,11]]},"reference":[{"key":"pdig.0001667.ref001","doi-asserted-by":"crossref","DOI":"10.3389\/fpsyt.2025.1609580","article-title":"How many people die by suicide each year? Not 727,000: a systematic review and meta-analysis of suicide underreporting across 71 countries over 122 years","volume":"16","author":"N Meda","year":"2025","journal-title":"Frontiers in Psychiatry"},{"key":"pdig.0001667.ref002","unstructured":"Australian Bureau of Statistics. Intentional self-harm (suicide) deaths. 2024."},{"issue":"2","key":"pdig.0001667.ref003","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0089944","article-title":"Hospital presenting self-harm and risk of fatal and non-fatal repetition: systematic review and meta-analysis","volume":"9","author":"R Carroll","year":"2014","journal-title":"PLOS ONE"},{"key":"pdig.0001667.ref004","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1192\/bjp.185.1.70","article-title":"Repetition of deliberate self-harm and subsequent suicide risk: long-term follow-up study of 11,583 patients","volume":"185","author":"DL Zahl","year":"2004","journal-title":"Br J Psychiatry"},{"issue":"1","key":"pdig.0001667.ref005","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1027\/0227-5910\/a000583","article-title":"Sentinel Surveillance for Self-Harm","volume":"40","author":"K Witt","year":"2019","journal-title":"Crisis"},{"issue":"24","key":"pdig.0001667.ref006","doi-asserted-by":"crossref","first-page":"9385","DOI":"10.3390\/ijerph17249385","article-title":"Development of a Self-Harm Monitoring System for Victoria","volume":"17","author":"J Robinson","year":"2020","journal-title":"Int J Environ Res Public Health"},{"issue":"8","key":"pdig.0001667.ref007","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1111\/1469-7610.00200","article-title":"Deliberate self-harm in adolescents: a study of characteristics and trends in Oxford, 1990-2000","volume":"44","author":"K Hawton","year":"2003","journal-title":"J Child Psychol Psychiatry"},{"key":"pdig.0001667.ref008","first-page":"17","article-title":"Definitions of suicidal behaviour","volume-title":"Suicidal behaviour: theories and research findings","author":"D De Leo","year":"2004"},{"key":"pdig.0001667.ref009","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195372922.001.0001","volume-title":"Principles & practice of public health surveillance","author":"LM Lee","year":"2010"},{"key":"pdig.0001667.ref010","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1002\/pds.2335","article-title":"A systematic review of validated methods for identifying suicide or suicidal ideation using administrative or claims data","author":"JT Walkup","year":"2012","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"2","key":"pdig.0001667.ref011","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0298411","article-title":"Validity and reliability International Classification of Diseases-10 codes for all forms of injury: A systematic review","volume":"19","author":"S Paleczny","year":"2024","journal-title":"PLoS One"},{"issue":"3","key":"pdig.0001667.ref012","doi-asserted-by":"crossref","DOI":"10.1002\/mpr.1559","article-title":"Emergency department and inpatient coding for self-harm and suicide attempts: Validation using clinician assessment data","volume":"26","author":"JR Randall","year":"2017","journal-title":"Int J Methods Psychiatr Res"},{"issue":"1","key":"pdig.0001667.ref013","first-page":"38","article-title":"Suicidal and self-harm presentations to Emergency Departments: The challenges of identification through diagnostic codes and presenting complaints","volume":"49","author":"J Sveticic","year":"2020","journal-title":"Health Inf Manag"},{"issue":"3","key":"pdig.0001667.ref014","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1093\/jamia\/ocab261","article-title":"Detection of self-harm and suicidal ideation in emergency department triage notes","volume":"29","author":"V Rozova","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"pdig.0001667.ref015","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1016\/j.auec.2023.01.003","article-title":"Characteristics of self-harm presentations to the emergency department of the Royal Melbourne Hospital, 2012\u20132019: Data from the Self-Harm Monitoring System for Victoria","volume":"26","author":"K Witt","year":"2023","journal-title":"Australasian Emergency Care"},{"issue":"9","key":"pdig.0001667.ref016","article-title":"The myth of generalisability in clinical research and machine learning in health care","volume":"2","author":"J Futoma","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"1","key":"pdig.0001667.ref017","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1038\/s41746-022-00690-x","article-title":"Moving towards vertically integrated artificial intelligence development","volume":"5","author":"J Zhang","year":"2022","journal-title":"npj Digital Medicine"},{"issue":"2","key":"pdig.0001667.ref018","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2019.20733","article-title":"Prospective and External Evaluation of a Machine Learning Model to Predict In-Hospital Mortality of Adults at Time of Admission","volume":"3","author":"N Brajer","year":"2020","journal-title":"JAMA Netw Open"},{"issue":"7","key":"pdig.0001667.ref019","article-title":"External validation of the myocardial-ischaemic-injury-index machine learning algorithm for the early diagnosis of myocardial infarction: a multicentre cohort study","volume":"6","author":"P Lopez-Ayala","year":"2024","journal-title":"Lancet Digit Health"},{"issue":"2","key":"pdig.0001667.ref020","doi-asserted-by":"crossref","DOI":"10.1056\/AIoa2300012","article-title":"Prospective Evaluation of Machine Learning for Public Health Screening: Identifying Unknown Hepatitis C Carriers","volume":"1","author":"N Dagan","year":"2024","journal-title":"NEJM AI"},{"issue":"3","key":"pdig.0001667.ref021","doi-asserted-by":"crossref","DOI":"10.1056\/AIoa2300099","article-title":"External Validation of a Commercial Acute Kidney Injury Predictive Model","volume":"1","author":"S Dutta","year":"2024","journal-title":"NEJM AI"},{"issue":"7","key":"pdig.0001667.ref022","article-title":"Epic Sepsis Model Inpatient Predictive Analytic Tool: A Validation Study","volume":"5","author":"J Cull","year":"2023","journal-title":"Crit Care Explor"},{"issue":"8","key":"pdig.0001667.ref023","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients","volume":"181","author":"A Wong","year":"2021","journal-title":"JAMA Intern Med"},{"issue":"3","key":"pdig.0001667.ref024","doi-asserted-by":"crossref","DOI":"10.1056\/AIoa2300032","article-title":"Evaluation of Sepsis Prediction Models before Onset of Treatment","volume":"1","author":"F Kamran","year":"2024","journal-title":"NEJM AI"},{"issue":"8","key":"pdig.0001667.ref025","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.29729","article-title":"Sepsis Prediction Model for Determining Sepsis vs SIRS, qSOFA, and SOFA","volume":"6","author":"AR Schertz","year":"2023","journal-title":"JAMA Netw Open"},{"issue":"4","key":"pdig.0001667.ref026","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1056\/NEJMp2311015","article-title":"Mind the Gap - Machine Learning, Dataset Shift, and History in the Age of Clinical Algorithms","volume":"390","author":"AS Lea","year":"2024","journal-title":"N Engl J Med"},{"issue":"5","key":"pdig.0001667.ref027","doi-asserted-by":"crossref","first-page":"986","DOI":"10.1093\/jamia\/ocx039","article-title":"Challenges in adapting existing clinical natural language processing systems to multiple, diverse health care settings","volume":"24","author":"DS Carrell","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001667.ref028","doi-asserted-by":"crossref","unstructured":"Elangovan A, et al. Principles from clinical research for NLP model generalization. In: Mexico City, Mexico, 2024.","DOI":"10.18653\/v1\/2024.naacl-long.127"},{"issue":"3","key":"pdig.0001667.ref029","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1093\/jamia\/ocx138","article-title":"Clinical documentation variations and NLP system portability: a case study in asthma birth cohorts across institutions","volume":"25","author":"S Sohn","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"pdig.0001667.ref030","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2018.5097","article-title":"Validation of Prediction Models for Critical Care Outcomes Using Natural Language Processing of Electronic Health Record Data","volume":"1","author":"BJ Marafino","year":"2018","journal-title":"JAMA Netw Open"},{"issue":"1","key":"pdig.0001667.ref031","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.artmed.2014.06.002","article-title":"Cross-hospital portability of information extraction of cancer staging information","volume":"62","author":"D Martinez","year":"2014","journal-title":"Artif Intell Med"},{"key":"pdig.0001667.ref032","doi-asserted-by":"crossref","first-page":"105115","DOI":"10.1016\/j.ijmedinf.2023.105115","article-title":"Generalizability and portability of natural language processing system to extract individual social risk factors","volume":"177","author":"T Magoc","year":"2023","journal-title":"Int J Med Inform"},{"key":"pdig.0001667.ref033","doi-asserted-by":"crossref","DOI":"10.1136\/amiajnl-2011-000583","article-title":"Portability of an algorithm to identify rheumatoid arthritis in electronic health records","volume":"19","author":"RJ Carroll","year":"2012","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"pdig.0001667.ref034","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1038\/s41746-021-00549-7","article-title":"Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review","volume":"5","author":"AAH de Hond","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"1","key":"pdig.0001667.ref035","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1186\/s12916-023-02779-w","article-title":"There is no such thing as a validated prediction model","volume":"21","author":"B Van Calster","year":"2023","journal-title":"BMC Med"},{"issue":"11","key":"pdig.0001667.ref036","doi-asserted-by":"crossref","first-page":"2686","DOI":"10.1038\/s41591-023-02540-z","article-title":"External validation of AI models in health should be replaced with recurring local validation","volume":"29","author":"A Youssef","year":"2023","journal-title":"Nat Med"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001667","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T17:43:01Z","timestamp":1789148581000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001667"}},"subtitle":[],"editor":[{"given":"Phat Kim","family":"Huynh","sequence":"first","affiliation":[{"name":"North Carolina A&T State University: North Carolina Agricultural and Technical State University","place":["UNITED STATES OF AMERICA"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,11]]},"references-count":36,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,11]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001667","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,11]]}}}