{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:15Z","timestamp":1785545055796,"version":"3.56.0"},"reference-count":24,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) and, more recently, large reasoning models (LRMs) have rapidly garnered significant interest for application in psychiatry and behavioral health. However, recent studies have identified significant shortcomings and potential risks in the performance of LLM-based systems, complicating their application to psychiatric diagnosis. Two promising approaches to addressing these challenges and improving the efficacy of these models are simulated reasoning (SR) and self-verification (SV), in which additional \u201creasoning tokens\u201d are used to guide model output, either during or after inference.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We aimed to explore how the use of SR (via LRMs) and SV (via supplemental prompting) affects the psychiatric diagnostic performance of LLMs.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n 106 case vignettes and associated diagnoses were extracted from the DSM-5-TR (Diagnostic and Statistical Manual, Version 5, Text Revision) Clinical Cases book, with permission. Both an LLM and an LRM model were selected from the latest available model generation for each of the two vendors studied (OpenAI and Google). Two inference approaches were developed: a Basic approach that directly prompted models to provide diagnoses and a SV approach that augmented the Basic approach with additional prompts. All case vignettes were processed by the two LLMs, two LRMs, and two inference approaches, and diagnostic performance was evaluated using the sensitivity and positive predictive value (PPV). Binomial generalized linear mixed models were used to test for significant differences between the model vendors (OpenAI, Google), type (LLM, LRM), and the addition of an SV prompt.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n All vignettes were successfully processed by each model and inference approach. Sensitivity ranged from 0.732 to 0.817, and PPV ranged from 0.534 to 0.779. The best overall performance was found in the\n o3-pro<\/jats:italic>\n LRM using SV, with a sensitivity of 0.782 and a PPV of 0.779. No statistically significant fixed effects were found for sensitivity. For PPV, a statistically significant effect was found for prompt type (SV,\n P<\/jats:italic>\n =.007) and model type (LRM,\n P<\/jats:italic>\n =.009). No significant interaction effects were identified.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n We found that both SR and SV yielded statistically significant improvements in the PPV, without significant differences in the sensitivity. The addition of the manually specified SV prompt improved the PPV even when simulated reasoning was used. This suggests that future efforts to apply language models in behavioral health could benefit from manually crafted reasoning prompts and automated SR.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/83927","type":"journal-article","created":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T18:11:11Z","timestamp":1780942271000},"page":"e83927-e83927","source":"Crossref","is-referenced-by-count":0,"title":["Simulated Reasoning and Self-Verification for Psychiatric Diagnosis in Generalist Large Language Models: Comparative Evaluation"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7442-9526","authenticated-orcid":false,"given":"Karthik V","family":"Sarma","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1462-4335","authenticated-orcid":false,"given":"Kaitlin E","family":"Hanss","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8943-5822","authenticated-orcid":false,"given":"Andrew J M","family":"Halls","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0739-1895","authenticated-orcid":false,"given":"Daniel F","family":"Becker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0690-9560","authenticated-orcid":false,"given":"Anne L","family":"Glowinski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6702-781X","authenticated-orcid":false,"given":"Andrew D","family":"Krystal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,8]]},"reference":[{"key":"R1","unstructured":"Villalobos P Ho A Sevilla J Besiroglu T Heim L Hobbhahn M . Position: will we run out of data? limits of LLM scaling based on human-generated data. Presented at: Proceedings of the 41st International Conference on Machine Learning Vienna; Jul 21-27, 2024:49523-49544; Vienna, Austria. [doi: 10.5555\/3692070.3694094]"},{"issue":"4","key":"R2","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2400802","article-title":"Randomized trial of a generative AI chatbot for mental health treatment","volume":"2","author":"Heinz","journal-title":"NEJM AI"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/69910","article-title":"Assessing the accuracy and reliability of large language models in psychiatry using standardized multiple-choice questions: cross-sectional study","volume":"27","author":"Hanss","journal-title":"J Med Internet Res"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2025.116844","article-title":"Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis","volume":"355","author":"Sarma","journal-title":"Psychiatry Res"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"301","DOI":"10.3390\/info16040301","article-title":"nBERT: harnessing NLP for emotion recognition in psychotherapy to transform mental health care","volume":"16","author":"Rasool","journal-title":"Information"},{"issue":"12","key":"R6","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1007\/s11920-023-01477-x","article-title":"An introduction to generative artificial intelligence in mental health care: considerations and guidance","volume":"25","author":"King","journal-title":"Curr Psychiatry Rep"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s44277-024-00010-z","article-title":"Opportunities and Risks of Large Language Models in Psychiatry","volume":"2","author":"Obradovich","journal-title":"NPP Digit Psychiatry Neurosci"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2024.116026","article-title":"Large language models in psychiatry: Opportunities and challenges","volume":"339","author":"Volkmer","journal-title":"Psychiatry Res"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijlp.2025.102086","article-title":"Large language models and psychiatry","volume":"101","author":"Orr\u00f9","journal-title":"Int J Law Psychiatry"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1176\/appi.ajp.2012.12070998","article-title":"DSM-5 Field Trials in the United States and Canada, Part I: study design, sampling strategy, implementation, and analytic approaches","volume":"170","author":"Clarke","journal-title":"AJP"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1176\/appi.ajp.2012.12070999","article-title":"DSM-5 field trials in the United States and Canada, Part II: test-retest reliability of selected categorical diagnoses","volume":"170","author":"Regier","journal-title":"Am J Psychiatry"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/75030","article-title":"Aiding large language models using clinical scoresheets for neurobehavioral diagnostic classification from text: algorithm development and validation","volume":"4","author":"Lin","journal-title":"JMIR AI"},{"key":"R13","unstructured":"Zhao C Tan Z Ma P et al. Is chain-of-thought reasoning of llms a mirage? A data distribution lens. arXiv. Preprint posted online on Aug 2, 2025. [doi: 10.48550\/arXiv.2508.01191]"},{"key":"R14","unstructured":"Chen Y Benton J Radhakrishnan A et al. Reasoning models don\u2019t always say what they think. arXiv. Preprint posted online on May 8, 2025. [doi: 10.48550\/arXiv.2505.05410]"},{"key":"R15","unstructured":"Alizadeh K Horton M Bengio S Farajtabar M . The illusion of thinking: understanding the strengths and limitations of reasoning models via the lens of problem complexity. 2025. URL: https:\/\/ml-site.cdn-apple.com\/papers\/the-illusion-of-thinking.pdf [Accessed 26-06-2025]"},{"key":"R16","unstructured":"OpenAI. Building more helpful ChatGPT experiences for everyone. 2025. URL: https:\/\/openai.com\/index\/building-more-helpful-chatgpt-experiences-for-everyone\/ [Accessed 10-09-2025]"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2026.4003","article-title":"Large language model performance and clinical reasoning tasks","volume":"9","author":"Rao","journal-title":"JAMA Netw Open"},{"issue":"10","key":"R18","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2500120","article-title":"Assessment of large language models in clinical reasoning: a novel benchmarking study","volume":"2","author":"McCoy","journal-title":"NEJM AI"},{"key":"R19","unstructured":"Arora RK Wei J Hicks RS et al. HealthBench: evaluating large language models towards improved human health. arXiv. Preprint posted online on May 13, 2025. [doi: 10.48550\/arXiv.2505.08775]"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-64769-1","article-title":"Quantifying the reasoning abilities of LLMs on clinical cases","volume":"16","author":"Qiu","journal-title":"Nat Commun"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/77699","article-title":"Large language models for psychiatric diagnosis based on multicenter real-world clinical records: comparative study","volume":"14","author":"Sun","journal-title":"JMIR Med Inform"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Lin CY Och FJ . Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. Presented at: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics; Jul 21-26, 2004:605-es; Barcelona, Spain. 2004.[doi: 10.3115\/1218955.1219032]","DOI":"10.3115\/1218955.1219032"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Papineni K Roukos S Ward T Zhu WJ . BLEU. Presented at: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics USA; Jul 7-12, 2002; Philadelphia, Pennsylvania. [doi: 10.3115\/1073083.1073135]","DOI":"10.3115\/1073083.1073135"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2025.104899","article-title":"Perplexity and proximity: Large language model perplexity complements semantic distance metrics for the detection of incoherent speech","volume":"170","author":"Xu","journal-title":"J Biomed Inform"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e83927","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:26:06Z","timestamp":1785536766000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e83927"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,8]]},"references-count":24,"URL":"https:\/\/doi.org\/10.2196\/83927","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,8]]},"article-number":"v5i4e83927"}}