{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T19:33:56Z","timestamp":1781379236397,"version":"3.54.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T00:00:00Z","timestamp":1700524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T00:00:00Z","timestamp":1700524800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Patients experiencing mental health crises often seek help through messaging-based platforms, but may face long wait times due to limited message triage capacity. Here we build and deploy a machine-learning-enabled system to improve response times to crisis messages in a large, national telehealth provider network. We train a two-stage natural language processing (NLP) system with key word filtering followed by logistic regression on 721 electronic medical record chat messages, of which 32% are potential crises (suicidal\/homicidal ideation, domestic violence, or non-suicidal self-injury). Model performance is evaluated on a retrospective test set (4\/1\/21\u20134\/1\/22, <jats:italic>N<\/jats:italic>\u2009=\u2009481) and a prospective test set (10\/1\/22\u201310\/31\/22, <jats:italic>N<\/jats:italic>\u2009=\u2009102,471). In the retrospective test set, the model has an AUC of 0.82 (95% CI: 0.78\u20130.86), sensitivity of 0.99 (95% CI: 0.96\u20131.00), and PPV of 0.35 (95% CI: 0.309\u20130.4). In the prospective test set, the model has an AUC of 0.98 (95% CI: 0.966\u20130.984), sensitivity of 0.98 (95% CI: 0.96\u20130.99), and PPV of 0.66 (95% CI: 0.626\u20130.692). The daily median time from message receipt to crisis specialist triage ranges from 8 to 13\u2009min, compared to 9\u2009h before the deployment of the system. We demonstrate that a NLP-based machine learning model can reliably identify potential crisis chat messages in a telehealth setting. Our system integrates into existing clinical workflows, suggesting that with appropriate training, humans can successfully leverage ML systems to facilitate triage of crisis messages.<\/jats:p>","DOI":"10.1038\/s41746-023-00951-3","type":"journal-article","created":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T19:04:29Z","timestamp":1700593469000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":42,"title":["Natural language processing system for rapid detection and intervention of mental health crisis chat messages"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3426-9289","authenticated-orcid":false,"given":"Akshay","family":"Swaminathan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iv\u00e1n","family":"L\u00f3pez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rafael Antonio Garcia","family":"Mar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tyler","family":"Heist","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom","family":"McClintock","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaitlin","family":"Caoili","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8063-7253","authenticated-orcid":false,"given":"Madeline","family":"Grace","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Rubashkin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael N.","family":"Boggs","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4387-8740","authenticated-orcid":false,"given":"Jonathan H.","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9965-5466","authenticated-orcid":false,"given":"Olivier","family":"Gevaert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Mou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew K.","family":"Nock","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,11,21]]},"reference":[{"key":"951_CR1","doi-asserted-by":"publisher","first-page":"1425","DOI":"10.3390\/ijerph15071425","volume":"15","author":"S Bachmann","year":"2018","unstructured":"Bachmann, S. Epidemiology of suicide and the psychiatric perspective. Int. J. Environ. Res. Public. Health 15, 1425 (2018).","journal-title":"Int. J. Environ. Res. Public. Health"},{"key":"951_CR2","unstructured":"Suicide worldwide in 2019. https:\/\/www.who.int\/publications-detail-redirect\/9789240026643."},{"key":"951_CR3","unstructured":"Key Substance Use and Mental Health Indicators in the United States: Results from the 2020 National Survey on Drug Use and Health. (2020)."},{"key":"951_CR4","doi-asserted-by":"crossref","unstructured":"Ehlman, D. C. Changes in Suicide Rates\u2014United States, 2019 and 2020. MMWR Morb. Mortal. Wkly. Rep. 71 (2022).","DOI":"10.15585\/mmwr.mm7108a5"},{"key":"951_CR5","unstructured":"2021 NAMI Impact Report. 11 https:\/\/www.nami.org\/Extranet\/NAMI-State-Organization-and-NAMI-Affiliate-Leaders\/NAMI-State-Organization-and-NAMI-Affiliate-Leaders\/Financial-and-Risk-Management\/2021-NAMI-Impact-Report_11-19-compress#:~:text=NAMI%27s%20HelpLine%20staff%20has%20doubled,help%20seekers%20monthly%20in%202019 (2021)."},{"key":"951_CR6","unstructured":"988 APPROPRIATIONS REPORT. https:\/\/www.samhsa.gov\/sites\/default\/files\/988-appropriations-report.pdf (2021)."},{"key":"951_CR7","unstructured":"Grant, R. Why Crisis Chat Rooms Are Making Victims Wait. The Atlantic https:\/\/www.theatlantic.com\/health\/archive\/2015\/07\/online-crisis-hotlines-chat-prevention\/398312\/ (2015)."},{"key":"951_CR8","first-page":"114176","volume":"283","author":"Z Xu","year":"2021","unstructured":"Xu, Z. et al. Detecting suicide risk using knowledge-aware natural language processing and counseling service data. Soc. Sci. Med. 1982 283, 114176 (2021).","journal-title":"Soc. Sci. Med. 1982"},{"key":"951_CR9","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1080\/10503307.2020.1781952","volume":"31","author":"N Bantilan","year":"2021","unstructured":"Bantilan, N., Malgaroli, M., Ray, B. & Hull, T. D. Just in time crisis response: suicide alert system for telemedicine psychotherapy settings. Psychother. Res. 31, 289\u2013299 (2021).","journal-title":"Psychother. Res."},{"key":"951_CR10","doi-asserted-by":"publisher","first-page":"8708434","DOI":"10.1155\/2016\/8708434","volume":"2016","author":"BL Cook","year":"2016","unstructured":"Cook, B. L. et al. Novel Use of Natural Language Processing (NLP) to predict suicidal ideation and psychiatric symptoms in a text-based mental health intervention in Madrid. Comput. Math. Methods Med. 2016, 8708434 (2016).","journal-title":"Comput. Math. Methods Med."},{"key":"951_CR11","unstructured":"Weiser, J. What we learned from training a machine learning model to detect suicidal risk. Medium https:\/\/research.crisistextline.org\/what-we-learned-from-training-a-machine-learning-model-to-detect-suicidal-risk-2c65f1d4d9eb (2021)."},{"key":"951_CR12","unstructured":"Womble, A., M. P. H., Affairs, V. P. of P., Torok, L. & Researcher, S. D. Everybody Hurts 2020. Crisis Text Line https:\/\/www.crisistextline.org\/blog\/2021\/04\/29\/everybody-hurts-2020\/ (2021)."},{"key":"951_CR13","doi-asserted-by":"crossref","unstructured":"ATA2023 Annual Conference & ExpoMarch 4\u20136, 2023San Antonio, Texas. Telemed. E-Health 29, A-1 (2023).","DOI":"10.1089\/tmj.2023.29089.abstracts"},{"key":"951_CR14","doi-asserted-by":"crossref","unstructured":"Seneviratne, M. G., Shah, N. H. & Chu, L. Bridging the implementation gap of machine learning in healthcare. BMJ Innov. 6, (2020).","DOI":"10.1136\/bmjinnov-2019-000359"},{"key":"951_CR15","doi-asserted-by":"publisher","first-page":"1115","DOI":"10.1213\/ANE.0000000000004575","volume":"130","author":"IS Hofer","year":"2020","unstructured":"Hofer, I. S., Burns, M., Kendale, S. & Wanderer, J. P. Realistically Integrating Machine Learning into clinical practice: a road map of opportunities, challenges, and a potential future. Anesth. Analg. 130, 1115\u20131118 (2020).","journal-title":"Anesth. Analg."},{"key":"951_CR16","doi-asserted-by":"publisher","first-page":"E1391","DOI":"10.1503\/cmaj.202066","volume":"193","author":"JP Cohen","year":"2021","unstructured":"Cohen, J. P. et al. Problems in the deployment of machine-learned models in health care. CMAJ Can. Med. Assoc. J. 193, E1391\u2013E1394 (2021).","journal-title":"CMAJ Can. Med. Assoc. J."},{"key":"951_CR17","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster, B. et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 17, 230 (2019).","journal-title":"BMC Med."},{"key":"951_CR18","unstructured":"APA Dictionary of Psychology. https:\/\/dictionary.apa.org\/."},{"key":"951_CR19","unstructured":"NCI Thesaurus. https:\/\/ncit.nci.nih.gov\/ncitbrowser\/."},{"key":"951_CR20","unstructured":"Domestic Violence. https:\/\/www.justice.gov\/ovw\/domestic-violence (2019)."},{"key":"951_CR21","doi-asserted-by":"publisher","unstructured":"Nock, M. K. & Favazza, A. R. Nonsuicidal self-injury: Definition and classification. in Understanding nonsuicidal self-injury: Origins, assessment, and treatment 9\u201318 (American Psychological Association, 2009). https:\/\/doi.org\/10.1037\/11875-001.","DOI":"10.1037\/11875-001"},{"key":"951_CR22","unstructured":"Ramos, J. E. Using TF-IDF to Determine Word Relevance in Document Queries. in (2003)."},{"key":"951_CR23","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1425-3","volume":"17","author":"L Wynants","year":"2019","unstructured":"Wynants, L. et al. Three myths about risk thresholds for prediction models. BMC Med. 17, 192 (2019).","journal-title":"BMC Med."},{"key":"951_CR24","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","volume":"26","author":"CJ Clopper","year":"1934","unstructured":"Clopper, C. J. & Pearson, E. S. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404\u2013413 (1934).","journal-title":"Biometrika"},{"key":"951_CR25","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","volume":"44","author":"ER DeLong","year":"1988","unstructured":"DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837\u2013845 (1988).","journal-title":"Biometrics"},{"key":"951_CR26","unstructured":"Piscopo, K. Suicidality and Death by Suicide Among Middle-aged Adults in the United States. https:\/\/www.samhsa.gov\/data\/sites\/default\/files\/report_3370\/ShortReport-3370.html (2017)."},{"key":"951_CR27","unstructured":"Unlock your productivity potential with Slack Platform | Slack. https:\/\/api.slack.com\/."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00951-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00951-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00951-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T19:20:41Z","timestamp":1700594441000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00951-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,21]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["951"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00951-3","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,21]]},"assertion":[{"value":"7 June 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 October 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The following authors report receiving a salary and owning equity in Cerebral Inc.: A.S., T.H., T.M., M.G., M.R., M.B., D.M. The following authors report part-time employment at Cerebral Inc. during the time of the study: I.L., R.A.G.M., K.C., A.S. owns stock in Roche (RHHVF) and is a consultant to Conduce Health. M.K.N. receives publication royalties from Macmillan, Pearson, and UpToDate. He has been a paid consultant in the past three years for Apple, Microsoft, and COMPASS Pathways, and for legal cases regarding a death by suicide. He has stock options in Cerebral Inc. He is an unpaid scientific advisor for Empatica, Koko, and TalkLife. J.H.C. reported receiving grants from the NIH\/National Institute on Drug Abuse Clinical Trials Network (UG1DA015815\u2013CTN-0136), Stanford Artificial Intelligence in Medicine and Imaging\u2013 Human-Centered Artificial Intelligence Partnership Grant, Doris Duke Charitable Foundation - Covid-19 Fund to Retain Clinical Scientists (20211260), Google Inc (in a research collaboration to leverage health data to predict clinical outcomes), and the American Heart Association - Strategically Focused Research Network - Diversity in Clinical Trials. J.H.C. reported receiving consulting fees from Sutton Pierce and Younker Hyde MacFarlane PLLC and being a co-founder of Reaction Explorer LLC, a company that develops and licenses organic chemistry education software using rule-based artificial intelligence technology. D.M. reports owning equity in Valera Inc. He is a consultant to Janssen Pharmaceuticals, Osmind, and COMPASS Pathway.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"213"}}