{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:31Z","timestamp":1785545071193,"version":"3.56.0"},"reference-count":27,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Language, which is a crucial element of human communication, is influenced by the complex interplay between thoughts, emotions, and experiences. Psychiatric disorders have an impact on cognitive and emotional processes, which in turn affect the content and way individuals with these disorders communicate using language. The recent rapid advancements in large language models (LLMs) suggest that leveraging them for quantitative analysis of language usage has the potential to become a useful method for providing objective measures in diagnosing and monitoring psychiatric conditions by analyzing language patterns.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to explore the use of LLMs in analyzing spontaneous communication to differentiate between various psychiatric disorders. We seek to show that the latent LLM embedding space identifies distinct linguistic markers that can be used to classify spontaneous communication from 7 different psychiatric disorders.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We used embeddings from the 7 billion parameter Generative Representational Instruction Tuning Language Model to analyze more than 37,000 posts from subreddits dedicated to seven common conditions: schizophrenia, borderline personality disorder (BPD), depression, attention-deficit\/hyperactivity disorder (ADHD), anxiety, posttraumatic stress disorder (PTSD) and bipolar disorder. A cross-validated multiclass Extreme Gradient Boosting classifier was trained on these embeddings to predict the origin subreddit for each post. Performance was evaluated using metrics such as precision, recall,\n F<\/jats:italic>\n 1-<\/jats:sub>\n score, and area under the receiver operating characteristic curve (AUC). In addition, we used Uniform Manifold Approximation and Projection dimensionality reduction to visualize relationships in language between these psychiatric disorders.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The 10-fold cross-validated Extreme Gradient Boosting classifier achieved a support-weighted average precision, recall,\n F<\/jats:italic>\n 1<\/jats:sub>\n , and accuracy score of 0.73, 0.73, 0.73, and 0.73, respectively. In one-versus-rest tasks, individual category AUCs ranged from 0.89 to 0.97, with a microaverage AUC of 0.95. ADHD posts were classified with the highest AUC of 0.97, indicating distinct linguistic features, while BPD posts had the lowest AUC of 0.89, suggesting greater linguistic overlap with other conditions. Consistent with the classifier results, the ADHD posts have a more visually distinct cluster in the Uniform Manifold Approximation and Projection projects, while BPD overlaps with depression, anxiety, and schizophrenia. Comparisons with other state-of-the-art embedding methods, such as OpenAI\u2019s text-embedding-3-small (AUC=0.94) and sentence-bidirectional encoder representations from transformers (AUC=0.86), demonstrated superior performance of the Generative Representational Instruction Tuning Language Model-7B model.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study introduces an innovative use of LLMs in psychiatry, showcasing their potential to objectively examine language use for distinguishing between different psychiatric disorders. The findings highlight the capability of LLMs to offer valuable insights into the linguistic patterns unique to various conditions, paving the way for more efficient, patient-focused diagnostic and monitoring strategies. Future research should aim to validate these results with clinically confirmed populations and investigate the implications of comorbidity and spectrum disorders.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/67369","type":"journal-article","created":{"date-parts":[[2025,5,30]],"date-time":"2025-05-30T17:01:22Z","timestamp":1748624482000},"page":"e67369-e67369","source":"Crossref","is-referenced-by-count":3,"title":["Algorithmic Classification of Psychiatric Disorder\u2013Related Spontaneous Communication Using Large Language Model Embeddings: Algorithm Development and Validation"],"prefix":"10.2196","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4310-9714","authenticated-orcid":false,"given":"Ryan Allen","family":"Shewcraft","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2490-5002","authenticated-orcid":false,"given":"John","family":"Schwarz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-0960-4561","authenticated-orcid":false,"given":"Mariann","family":"Micsinai Balan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,5,30]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2020.00076","article-title":"The heterogeneity of mental health assessment","volume":"11","author":"Newson","journal-title":"Front Psychiatry"},{"issue":"6","key":"R2","first-page":"24","volume":"6","author":"Aboraya","journal-title":"Psychiatry (Edgmont)"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1038\/npjschz.2015.30","article-title":"Automated analysis of free speech predicts psychosis onset in high-risk youths","volume":"1","author":"Bedi","journal-title":"NPJ Schizophr"},{"issue":"3","key":"R4","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1521\/pedi.2012.26.3.305","article-title":"Expressive language disturbance in borderline personality disorder in response to emotional autobiographical stimuli","volume":"26","author":"Carter","journal-title":"J Pers Disord"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"870","DOI":"10.1038\/s41467-022-28513-3","article-title":"Using language in social media posts to study the network dynamics of depression longitudinally","volume":"13","author":"Kelley","journal-title":"Nat Commun"},{"issue":"5","key":"R6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0251787","article-title":"The relationship between linguistic expression in blog content and symptoms of depression, anxiety, and suicidal thoughts: A longitudinal study","volume":"16","author":"O\u2019Dea","journal-title":"PLoS ONE"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.3390\/brainsci13010028","article-title":"Artificial intelligence-enabled end-to-end detection and assessment of Alzheimer\u2019s disease using voice","volume":"13","author":"Agbavor","journal-title":"Brain Sci"},{"issue":"12","key":"R8","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000168","article-title":"Predicting dementia from spontaneous speech using large language models","volume":"1","author":"Agbavor","journal-title":"PLOS Digit Health"},{"key":"R9","unstructured":"Muennighoff N Su H Wang L et al. Generative representational instruction tuning. arXiv. Preprint posted online on Feb 15, 2024. [doi: 10.48550\/arXiv.2402.09906]"},{"issue":"10","key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/22635","article-title":"Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on Reddit during COVID-19: observational study","volume":"22","author":"Low","journal-title":"J Med Internet Res"},{"key":"R11","unstructured":"Reddit mental health dataset. Zendodo. Jul 13, 2020. URL: https:\/\/zenodo.org\/records\/3941387 [Accessed 04-04-2024]"},{"key":"R12","unstructured":"Jiang AQ Sablayrolles A Mensch A et al. Mistral 7B. arXiv. Preprint posted online on Oct 10, 2023. [doi: 10.48550\/arXiv.2310.06825]"},{"key":"R13","unstructured":"New embedding models and API updates. OpenAI. Jan 25, 2024. URL: https:\/\/openai.com\/index\/new-embedding-models-and-api-updates [Accessed 26-05-2025]"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Reimers N Gurevych I . Sentence-BERT: sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. Association for Computational Linguistics; 2019:3982-3992. [doi: 10.18653\/v1\/D19-1410]","DOI":"10.18653\/v1\/D19-1410"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbi.2019.10.006","article-title":"Appetite change profiles in depression exhibit differential relationships between systemic inflammation and activity in reward and interoceptive neurocircuitry","volume":"83","author":"Cosgrove","journal-title":"Brain Behav Immun"},{"issue":"4","key":"R16","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1016\/s0733-8619(05)70283-8","article-title":"Sleep in psychiatric disorders","volume":"14","author":"Benca","journal-title":"Neurol Clin"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0182-1","article-title":"GPS mobility as a digital biomarker of negative symptoms in schizophrenia: a case control study","volume":"2","author":"Depp","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/j.brat.2009.09.007","article-title":"Word use of outpatients with a personality disorder and concurrent or previous major depressive disorder","volume":"48","author":"Molendijk","journal-title":"Behav Res Ther"},{"issue":"12","key":"R19","doi-asserted-by":"publisher","first-page":"1733","DOI":"10.1176\/ajp.155.12.1733","article-title":"Axis I comorbidity of borderline personality disorder","volume":"155","author":"Zanarini","journal-title":"Am J Psychiatry"},{"key":"R20","unstructured":"National Collaborating Centre for Mental Health (UK). Borderline Personality Disorder: Treatment and Management. British Psychological Society (UK); 2009."},{"issue":"1","key":"R21","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/j.jad.2005.03.005","article-title":"Comorbid psychiatric disorders in depressed outpatients: demographic and clinical features","volume":"87","author":"Rush","journal-title":"J Affect Disord"},{"key":"R22","volume":"61 Suppl 7","author":"Brady","journal-title":"J Clin Psychiatry"},{"issue":"3","key":"R23","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1192\/bjp.bp.106.030957","article-title":"The bipolar spectrum","volume":"190","author":"Angst","journal-title":"Br J Psychiatry"},{"issue":"1-2","key":"R24","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/s0165-0327(97)00057-8","article-title":"The depressive spectrum: diagnostic classification and course","volume":"45","author":"Angst","journal-title":"J Affect Disord"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1016\/s0193-953x(05)70093-9","article-title":"The evolving bipolar spectrum. Prototypes I, II, III, and IV","volume":"22","author":"Akiskal","journal-title":"Psychiatr Clin North Am"},{"issue":"6","key":"R26","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1177\/1745691613504115","article-title":"636,120 ways to have posttraumatic stress disorder","volume":"8","author":"Galatzer-Levy","journal-title":"Perspect Psychol Sci"},{"key":"R27","unstructured":"Benara V Singh C Morris JX et al. Crafting interpretable embeddings by asking LLMs questions. arXiv. Preprint posted online on May 26, 2024. [doi: 10.48550\/arXiv.2405.16714]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e67369","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:28:00Z","timestamp":1785536880000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e67369"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,30]]},"references-count":27,"URL":"https:\/\/doi.org\/10.2196\/67369","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,30]]},"article-number":"v4i6e67369"}}