{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T23:49:02Z","timestamp":1791503342672,"version":"4.3.4"},"reference-count":24,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Ment Health"],"abstract":"\n Background<\/jats:title>\n Mental illness contributes substantially to global disability, and public adoption of AI for mental health support is accelerating without commensurate safety evaluation. General-purpose large language models can fail to recognize naturalistic text expressing suicidal ideation, with immediate clinical consequences. Domain-specific natural language processing methods offer a contrasting approach, but ensembles integrating the two have not been formally characterized.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We aimed to (1) benchmark 5 modeling approaches for classifying mental health\u2013related social media text, (2) develop and optimize ensembles integrating a domain-specific natural language processing classifier with a fine-tuned large language model, and (3) derive a closed-form framework constraining ensemble weighting in safety-critical classification.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We analyzed 60,889 publicly available Reddit and Twitter texts spanning 9 categories (anxiety, bipolar disorder, depression, a normal baseline, personality disorder, stress, suicidal ideation, attention-deficit\/hyperactivity disorder, and autism spectrum disorder). Labels were derived from the originating subreddit or a self-stated condition and are not clinical diagnoses. Five architectures were compared: prompt-engineered base GPT-4o-mini; fine-tuned GPT-4o-mini; a domain-specific classifier (support vector machine over symptom-informed lexical features); and 2 ensembles of these models, hybrid probability\u2013indicator fusion and soft probability fusion. Data were split into 70%, 10%, and 20% at the post level (n=42,622, n=6089, and n=12,178); weights were selected on the validation split and all reported performance comes from the held-out test split. Closed-form bounds on permissible language model weights were derived for both strategies. CIs are paired bootstrap intervals (n=10,000 replicates); accuracy differences were tested with exact McNemar tests.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The domain-specific classifier achieved 91.9% accuracy (95% CI 91.4%-92.4%), exceeding the base (58.7%) and fine-tuned (88.6%) large language models. Both ensembles outperformed either constituent model, reaching 93.9% (soft fusion; 95% CI 93.5%-94.3%) and 93.8% (hybrid fusion; 95% CI 93.4%-94.2%) at a weighting of 60% classifier to 40% fine-tuned model, selected on validation (P<.001 for both against the classifier alone; the 2 ensembles did not differ from each other [P=.16]). The hybrid ensemble reduced the miss rate for the suicidal ideation category from 32.5% (650\/2000) to 1.9% (37\/2000), a 17.6-fold reduction relative to the base model, although its advantage over the classifier alone within that category was not significant (P=.68). Accuracy fell discontinuously above a 50% language model weight, where the hybrid ensemble reduced exactly to the fine-tuned model; the selected weight satisfies the derived instance-level bound of 44.0%.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Domain-specific clinical grounding remained necessary for stable ensemble performance in this corpus, and probabilistic fusion outperformed both constituent models when language model weight was bounded by a derivable, model-agnostic constraint. Because labels were community-inferred and only one corpus was analyzed, these results are hypotheses requiring external validation on clinically characterized data before use in decision support.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/98925","type":"journal-article","created":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T22:50:09Z","timestamp":1791499809000},"page":"e98925","source":"Crossref","is-referenced-by-count":0,"title":["Ensemble of Domain-Specific Natural Language Processing and Large Language Models for Detecting Suicidal Ideation and Mental Health Conditions in Social Media Text: Development and Evaluation Study"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2344-4133","authenticated-orcid":false,"given":"Thomas","family":"Kallstenius","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5429-5594","authenticated-orcid":false,"given":"Basil","family":"Duvernoy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9662-7550","authenticated-orcid":false,"given":"Adam","family":"Kallstenius","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1758-2206","authenticated-orcid":false,"given":"Andrea Johansson","family":"Capusan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5632-0084","authenticated-orcid":false,"given":"Adam","family":"Williamson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,8]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1016\/s0140-6736(13)61611-6"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1016\/s2215-0366(15)00505-2"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1016\/j.glmedi.2024.100099"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2024.1280235"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1017\/S0033291724003295"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.2196\/16021"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.2196\/43862"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.2196\/12106"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2017.04.046"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1002\/wps.20673"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-021-01093-w"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1002\/wps.20883"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.2471\/BLT.19.237636"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.2196\/60589"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2019.103978"},{"key":"ref17","author":"Moore, J","year":"2025","journal-title":"FAccT '25: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2509.24857"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.2196\/67891"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ps.20250086"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-68764-y"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-08031-0"},{"key":"ref24","unstructured":"Claude2026-10-01https:\/\/claude.ai\/new"}],"container-title":["JMIR Mental Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mental.jmir.org\/2026\/1\/e98925","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T22:50:10Z","timestamp":1791499810000},"score":1,"resource":{"primary":{"URL":"https:\/\/mental.jmir.org\/2026\/1\/e98925"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,8]]},"references-count":24,"URL":"https:\/\/doi.org\/10.2196\/98925","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.98925","asserted-by":"object"}]},"ISSN":["2368-7959"],"issn-type":[{"value":"2368-7959","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,8]]}}}