{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T02:25:40Z","timestamp":1772504740916,"version":"3.50.1"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T00:00:00Z","timestamp":1681257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Garvey Institute for Brain Health Solutions Innovation"},{"name":"Informatics-Supported Authorship for Caring"},{"name":"Military Suicide Research Consortiu"},{"name":"Office of the Assistant Secretary of Defense for Health Affairs","award":["W81XWH-10-2-0181"],"award-info":[{"award-number":["W81XWH-10-2-0181"]}]},{"DOI":"10.13039\/100000005","name":"Department of Defense","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100020960","name":"Military Suicide Research Consortium","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100020960","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,5,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Compared to natural language processing research investigating suicide risk prediction with social media (SM) data, research utilizing data from clinical settings are scarce. However, the utility of models trained on SM data in text from clinical settings remains unclear. In addition, commonly used performance metrics do not directly translate to operational value in a real-world deployment. The objectives of this study were to evaluate the utility of SM-derived training data for suicide risk prediction in a clinical setting and to develop a metric of the clinical utility of automated triage of patient messages for suicide risk.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Using clinical data, we developed a Bidirectional Encoder Representations from Transformers-based suicide risk detection model to identify messages indicating potential suicide risk. We used both annotated and unlabeled suicide-related SM posts for multi-stage transfer learning, leveraging customized contemporary learning rate schedules. We also developed a novel metric estimating predictive models\u2019 potential to reduce follow-up delays with patients in distress and used it to assess model utility.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Multi-stage transfer learning from SM data outperformed baseline approaches by traditional classification performance metrics, improving performance from 0.734 to a best F1 score of 0.797. Using this approach for automated triage could reduce response times by 15 minutes per urgent message.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Despite differences in data characteristics and distribution, publicly available SM data benefit clinical suicide risk prediction when used in conjunction with contemporary transfer learning techniques. Estimates of time saved due to automated triage indicate the potential for the practical impact of such models when deployed as part of established suicide prevention interventions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>This work demonstrates a pathway for leveraging publicly available SM data toward improving risk assessment, paving the way for better clinical care and improved clinical outcomes.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad062","type":"journal-article","created":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T20:37:13Z","timestamp":1681331833000},"page":"1068-1078","source":"Crossref","is-referenced-by-count":14,"title":["From benchmark to bedside: transfer learning from social media to patient-provider text messages for suicide risk prediction"],"prefix":"10.1093","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9386-5569","authenticated-orcid":false,"given":"Hannah A","family":"Burkhardt","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, Washington, USA"}]},{"given":"Xiruo","family":"Ding","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, Washington, USA"}]},{"given":"Amanda","family":"Kerbrat","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Sciences, University of Washington , Seattle, Washington, USA"}]},{"given":"Katherine Anne","family":"Comtois","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Sciences, University of Washington , Seattle, Washington, USA"}]},{"given":"Trevor","family":"Cohen","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Medical Education, University of Washington , Seattle, Washington, USA"}]}],"member":"286","published-online":{"date-parts":[[2023,4,12]]},"reference":[{"issue":"22","key":"2024041200123982600_ocad062-B1","doi-asserted-by":"crossref","first-page":"617","DOI":"10.15585\/mmwr.mm6722a1","article-title":"Trends in state suicide rates 1999-2016","volume":"67","author":"Stone","year":"2018","journal-title":"MMWR Morb Mortal Wkly Rep"},{"issue":"3","key":"2024041200123982600_ocad062-B2","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1176\/appi.ps.201800381","article-title":"Self-reported suicidal ideation as a predictor of suicidal behavior among outpatients with diagnoses of psychotic disorders","volume":"70","author":"Simon","year":"2019","journal-title":"Psychiatr Serv"},{"key":"2024041200123982600_ocad062-B3","first-page":"1","author":"The Joint Commission","year":"2019"},{"issue":"9","key":"2024041200123982600_ocad062-B4","doi-asserted-by":"crossref","first-page":"894","DOI":"10.1001\/jamapsychiatry.2018.1776","article-title":"Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department","volume":"75","author":"Stanley","year":"2018","journal-title":"JAMA Psychiatry"},{"issue":"5","key":"2024041200123982600_ocad062-B5","doi-asserted-by":"crossref","first-page":"474","DOI":"10.1001\/jamapsychiatry.2018.4530","article-title":"Effect of augmenting standard care for military personnel with brief caring text messages for suicide prevention","volume":"76","author":"Comtois","year":"2019","journal-title":"JAMA Psychiatry"},{"issue":"5","key":"2024041200123982600_ocad062-B6","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1037\/pro0000134","article-title":"Implementation methods for the caring contacts suicide prevention intervention","volume":"48","author":"Reger","year":"2017","journal-title":"Prof Psychol Res Pract"},{"issue":"1","key":"2024041200123982600_ocad062-B7","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1037\/ser0000645","article-title":"Caring contacts for suicide prevention: a systematic review and meta-analysis","volume":"20","author":"Skopp","year":"2022","journal-title":"Psychol Serv"},{"key":"2024041200123982600_ocad062-B8","author":"Burkhardt","year":"2022"},{"issue":"2","key":"2024041200123982600_ocad062-B9","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MIS.2009.36","article-title":"The unreasonable effectiveness of data","volume":"24","author":"Halevy","year":"2009","journal-title":"IEEE Intell Syst"},{"key":"2024041200123982600_ocad062-B10","doi-asserted-by":"publisher","first-page":"8342","DOI":"10.18653\/v1\/2020.acl-main.740","author":"Gururangan","year":"2020"},{"issue":"5","key":"2024041200123982600_ocad062-B11","doi-asserted-by":"crossref","first-page":"1102","DOI":"10.1093\/jamia\/ocv066","article-title":"Report of the AMIA EHR-2020 Task Force on the status and future direction of EHRs","volume":"22","author":"Payne","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024041200123982600_ocad062-B12","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.hjdsi.2013.12.005","article-title":"Health information exchange among U.S. hospitals: Who\u2019s in, who\u2019s out, and why?","volume":"2","author":"Adler-Milstein","year":"2014","journal-title":"Healthc (Amst)"},{"key":"2024041200123982600_ocad062-B13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/1178222618792860","article-title":"Natural language processing of social media as screening for suicide risk","volume":"10","author":"Coppersmith","year":"2018","journal-title":"Biomed Inform Insights"},{"key":"2024041200123982600_ocad062-B14","doi-asserted-by":"publisher","first-page":"25","DOI":"10.18653\/v1\/W18-0603","volume-title":"proceedings of the fifth workshop on computational linguistics and clinical psychology: from keyboard to clinic","author":"Shing","year":"2018"},{"key":"2024041200123982600_ocad062-B15","first-page":"24","author":"Zirikly","year":"2019"},{"issue":"1","key":"2024041200123982600_ocad062-B16","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1055\/s-0041-1726522","article-title":"A review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records","volume":"30","author":"Laparra","year":"2021","journal-title":"Yearb Med Inform"},{"key":"2024041200123982600_ocad062-B17","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-016-0043-6","volume-title":"A Survey of Transfer Learning","author":"Weiss","year":"2016"},{"key":"2024041200123982600_ocad062-B18","author":"Naseem","year":"2022"},{"key":"2024041200123982600_ocad062-B19","author":"Ji","year":"2021"},{"issue":"4","key":"2024041200123982600_ocad062-B20","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","journal-title":"Bioinformatics"},{"key":"2024041200123982600_ocad062-B21","doi-asserted-by":"publisher","first-page":"328","DOI":"10.18653\/v1\/p18-1031","author":"Howard","year":"2018"},{"issue":"2","key":"2024041200123982600_ocad062-B22","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1136\/bmjinnov-2019-000359","article-title":"Bridging the implementation gap of machine learning in healthcare","volume":"6","author":"Seneviratne","year":"2020","journal-title":"BMJ Innov"},{"issue":"9","key":"2024041200123982600_ocad062-B23","doi-asserted-by":"crossref","first-page":"e2233946","DOI":"10.1001\/jamanetworkopen.2022.33946","article-title":"Randomized Clinical Trials of Machine Learning Interventions in Health Care","volume":"5","author":"Plana","year":"2022","journal-title":"JAMA Netw Open"},{"issue":"6","key":"2024041200123982600_ocad062-B24","doi-asserted-by":"crossref","first-page":"1353","DOI":"10.1093\/jamia\/ocab035","article-title":"Conflicting information from the Food and Drug Administration: missed opportunity to lead standards for safe and effective medical artificial intelligence solutions","volume":"28","author":"Hernandez-Boussard","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2024041200123982600_ocad062-B25","doi-asserted-by":"crossref","first-page":"1149","DOI":"10.1093\/jamia\/ocaa318","article-title":"A framework for making predictive models useful in practice","volume":"28","author":"Jung","year":"2021","journal-title":"J Am Med Informatics Assoc"},{"issue":"10","key":"2024041200123982600_ocad062-B26","doi-asserted-by":"crossref","first-page":"e109264","DOI":"10.1371\/journal.pone.0109264","article-title":"Data-driven decisions for reducing readmissions for heart failure: general methodology and case study","volume":"9","author":"Bayati","year":"2014","journal-title":"PLoS One"},{"key":"2024041200123982600_ocad062-B27","author":"Barthel","year":"2016"},{"issue":"1","key":"2024041200123982600_ocad062-B28","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1609\/icwsm.v7i1.14443","article-title":"Dude, srsly?: The surprisingly formal nature of Twitter\u2019s language","volume":"7","author":"Hu","year":"2021","journal-title":"ICWSM"},{"key":"2024041200123982600_ocad062-B29","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1145\/2998181.2998220","author":"De Choudhury","year":"2017"},{"key":"2024041200123982600_ocad062-B30","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","author":"Devlin","year":"2019"},{"key":"2024041200123982600_ocad062-B31","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009"},{"key":"2024041200123982600_ocad062-B32","first-page":"2079","article-title":"On over-fitting in model selection and subsequent selection bias in performance evaluation","volume":"11","author":"Cawley","year":"2010","journal-title":"J Mach Learn Res"},{"key":"2024041200123982600_ocad062-B33","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2024041200123982600_ocad062-B34","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","author":"Wolf","year":"2019"},{"key":"2024041200123982600_ocad062-B35","doi-asserted-by":"publisher","first-page":"51","DOI":"10.3115\/v1\/W14-3207","volume-title":"proceedings of the workshop on computational linguistics and clinical psychology: from linguistic signal to clinical reality","author":"Coppersmith","year":"2014"},{"key":"2024041200123982600_ocad062-B36","doi-asserted-by":"publisher","first-page":"58","DOI":"10.18653\/v1\/W17-3107","author":"Shen","year":"2017"},{"key":"2024041200123982600_ocad062-B37","doi-asserted-by":"publisher","author":"De Choudhury","year":"2014","DOI":"10.1145\/2531602.2531675"},{"key":"2024041200123982600_ocad062-B38","first-page":"1348","author":"Resnik","year":"2013"},{"issue":"4","key":"2024041200123982600_ocad062-B39","doi-asserted-by":"crossref","first-page":"669","DOI":"10.1007\/s13347-018-0336-0","article-title":"Ethics and artificial intelligence: suicide prevention on Facebook","volume":"31","author":"Gomes de Andrade","year":"2018","journal-title":"Philos Technol"},{"issue":"9958","key":"2024041200123982600_ocad062-B40","doi-asserted-by":"crossref","first-page":"1917","DOI":"10.1016\/S0140-6736(14)62267-4","article-title":"Trouble on the radar","volume":"384","author":"Lee","year":"2014","journal-title":"Lancet"},{"issue":"1","key":"2024041200123982600_ocad062-B41","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1186\/s12888-020-02721-x","article-title":"Two-way messaging therapy for depression and anxiety: longitudinal response trajectories","volume":"20","author":"Hull","year":"2020","journal-title":"BMC Psychiatry"},{"key":"2024041200123982600_ocad062-B42","doi-asserted-by":"publisher","first-page":"105","DOI":"10.18653\/v1\/2022.clpsych-1.9","author":"Burkhardt","year":"2022"},{"issue":"8","key":"2024041200123982600_ocad062-B43","doi-asserted-by":"publisher","first-page":"565","DOI":"10.7326\/M19-0366","article-title":"Ethics, transparency, and public health at the intersection of innovation and Facebook\u2019s suicide prevention efforts","volume":"170","author":"Barnett","year":"2019","journal-title":"Ann Intern Med"},{"key":"2024041200123982600_ocad062-B44","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1145\/3287560.3287587","author":"Chancellor","year":"2019"},{"issue":"6245","key":"2024041200123982600_ocad062-B45","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1126\/science.aac4520","article-title":"Data, privacy, and the greater good","volume":"349","author":"Horvitz","year":"2015","journal-title":"Science"},{"key":"2024041200123982600_ocad062-B46","author":"Singer","year":"2018"},{"issue":"4","key":"2024041200123982600_ocad062-B47","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1111\/jcom.12106","article-title":"Self-disclosure in social media: extending the functional approach to disclosure motivations and characteristics on social network sites","volume":"64","author":"Bazarova","year":"2014","journal-title":"J Commun"},{"issue":"2","key":"2024041200123982600_ocad062-B48","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1177\/0261927X12456384","article-title":"Managing impressions and relationships on Facebook: self-presentational and relational concerns revealed through the analysis of language style","volume":"32","author":"Bazarova","year":"2013","journal-title":"J Lang Soc Psychol"},{"key":"2024041200123982600_ocad062-B49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.copsyc.2019.06.025","article-title":"Self-disclosure versus self-presentation on social media","volume":"31","author":"Schlosser","year":"2020","journal-title":"Curr Opin Psychol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/6\/1068\/50374845\/ocad062.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/6\/1068\/50374845\/ocad062.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,12]],"date-time":"2024-04-12T00:13:06Z","timestamp":1712880786000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/6\/1068\/7116301"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,12]]},"references-count":49,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,4,12]]},"published-print":{"date-parts":[[2023,5,19]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad062","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,6,1]]},"published":{"date-parts":[[2023,4,12]]}}}