{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:23:00Z","timestamp":1792790580609,"version":"4.5.0"},"reference-count":37,"publisher":"American Medical Association (AMA)","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n Large language models (LLMs) are increasingly applied to mental health contexts, yet their capacity to generate responses that align with evidence-based psychotherapy remains uncertain. Motivational interviewing (MI), a structured counseling approach, provides an empirically grounded setting for evaluating alignment between LLM-generated and human therapist responses.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To evaluate how closely an LLM\u2019s responses align with therapist responses in MI sessions, using automated similarity metrics.<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n This cross-sectional study used high-fidelity therapist-client transcripts annotated with the Motivational Interviewing Treatment Integrity system. Transcripts were sourced from publicly available counseling videos. For each therapist turn, the GPT-4o LLM generated a response using a standardized, MI-informed prompt based on the preceding conversation context. Analyses were conducted between March and May 2025.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Alignment between LLM-generated and therapist responses was assessed using (1) cosine similarity based on sentence embeddings to capture semantic overlap and (2) DeepEval, a contextual deep-learning\u2013based metric assessing coherence and contextual appropriateness. A therapist topic-consistency index quantified within-session thematic coherence and was examined as a moderator of alignment.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n A total of 3706 therapist turns from 154 MI sessions were evaluated. Mean (SD) DeepEval scores were higher than mean (SD) cosine similarity scores (0.72 [0.31] vs 0.29 [0.20];\n P<\/jats:italic>\n \u2009&amp;lt;\u2009.001), suggesting limited semantic overlap despite greater contextual appropriateness. Therapist topic consistency significantly moderated similarity, where cosine similarity was higher in high-consistency than low-consistency sessions (mean [SD] difference, 0.027 [0.007];\n t<\/jats:italic>\n 3706<\/jats:sub>\n \u2009=\u20093.987;\n P<\/jats:italic>\n \u2009&amp;lt;\u2009.001), as was DeepEval score (mean [SD] difference, 0.038 [0.010];\n t<\/jats:italic>\n 3706<\/jats:sub>\n \u2009=\u20093.747;\n P<\/jats:italic>\n \u2009&amp;lt;\u2009.001). Correlation between metrics was negligible (Spearman \u03c1, \u20130.01), indicating that they captured distinct aspects of response alignment. LLM performance declined slightly across longer conversations (mean [SD] slope reduction for cosine similarity, \u22120.0005 [0.0016], and for DeepEval, \u22120.0005 [0.0022]), with increased verbosity and signs of reduced contextual grounding.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n In this cross-sectional study of 154 MI sessions, prompted LLMs showed general alignment with therapist responses in MI-oriented conversations, as judged by automated similarity metrics. However, limitations in long-range coherence, stylistic alignment, and the use of indirect proxies for therapeutic quality highlight the need for improved prompt design, MI-specific evaluation methods, and clinical validation before integration into mental health care.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2026.2750","type":"journal-article","created":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T15:30:29Z","timestamp":1774279829000},"page":"e262750","source":"Crossref","is-referenced-by-count":8,"title":["Alignment of Large Language Model Responses With Human Therapists in Motivational Interviewing"],"prefix":"10.1001","volume":"9","author":[{"given":"Bazen Gashaw","family":"Teferra","sequence":"first","affiliation":[{"name":"Interventional Psychiatry Program, St Michael\u2019s Hospital, Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandra","family":"Huang","sequence":"additional","affiliation":[{"name":"David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nabil","family":"Johny","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, iBioMed Program, McMaster University, Hamilton, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Argyrios","family":"Perivolaris","sequence":"additional","affiliation":[{"name":"Interventional Psychiatry Program, St Michael\u2019s Hospital, Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huda","family":"Al-Shamali","sequence":"additional","affiliation":[{"name":"Interventional Psychiatry Program, St Michael\u2019s Hospital, Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karisa","family":"Parkington","sequence":"additional","affiliation":[{"name":"Interventional Psychiatry Program, St Michael\u2019s Hospital, Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alice","family":"Rueda","sequence":"additional","affiliation":[{"name":"Interventional Psychiatry Program, St Michael\u2019s Hospital, Unity Health Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard J.","family":"Zeifman","sequence":"additional","affiliation":[{"name":"Department of Psychology, The New School for Social Research, New York, New York"},{"name":"NYU Center for Psychedelic Medicine, NYU Grossman School of Medicine, New York, New York"},{"name":"Centre for Psychedelic Research, Department of Brain Sciences, Faculty of Medicine, Imperial College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Divya","family":"Sharma","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics, York University, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sri","family":"Krishnan","sequence":"additional","affiliation":[{"name":"Electrical, Computer, and Biomedical Engineering, Toronto Metropolitan University, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Candice","family":"Monson","sequence":"additional","affiliation":[{"name":"Department of Psychology, Toronto Metropolitan University, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Venkat","family":"Bhat","sequence":"additional","affiliation":[{"name":"Interventional Psychiatry Program, St Michael\u2019s Hospital, Unity Health Toronto, Toronto, Ontario, Canada"},{"name":"Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2026,3,23]]},"reference":[{"key":"zoi260119r1","doi-asserted-by":"publisher","DOI":"10.1016\/j.glmedi.2024.100099","article-title":"Enhancing mental health with artificial intelligence: current trends and future prospects.","volume":"3","author":"Olawade","year":"2024","journal-title":"J Med Surg Public Health"},{"issue":"1","key":"zoi260119r3","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1038\/s41746-025-01611-4","article-title":"A scoping review of large language models for generative tasks in mental health care.","volume":"8","author":"Hua","year":"2025","journal-title":"NPJ Digit Med"},{"key":"zoi260119r4","doi-asserted-by":"publisher","DOI":"10.2196\/55067","article-title":"Screening for depression using natural language processing: literature review.","volume":"13","author":"Teferra","year":"2024","journal-title":"Interact J Med Res"},{"key":"zoi260119r5","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2024.1280235","article-title":"Artificial intelligence in positive mental health: a narrative review.","volume":"6","author":"Thakkar","year":"2024","journal-title":"Front Digit Health"},{"key":"zoi260119r6","doi-asserted-by":"publisher","DOI":"10.2196\/44325","article-title":"Predicting generalized anxiety disorder from impromptu speech transcripts using context-aware transformer-based neural networks: model evaluation study.","volume":"10","author":"Teferra","year":"2023","journal-title":"JMIR Ment Health"},{"key":"zoi260119r7","doi-asserted-by":"publisher","DOI":"10.2196\/57400","article-title":"Large language models for mental health applications: systematic review.","volume":"11","author":"Guo","year":"2024","journal-title":"JMIR Ment Health"},{"issue":"1","key":"zoi260119r8","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/365153.365168","article-title":"ELIZA\u2014a computer program for the study of natural language communication between man and machine.","volume":"9","author":"Weizenbaum","year":"1966","journal-title":"Commun ACM"},{"issue":"2","key":"zoi260119r9","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785","article-title":"Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial.","volume":"4","author":"Fitzpatrick","year":"2017","journal-title":"JMIR Ment Health"},{"issue":"3","key":"zoi260119r10","doi-asserted-by":"publisher","DOI":"10.2196\/36910","article-title":"Delivery of a mental health intervention for chronic pain through an artificial intelligence-enabled app (Wysa): protocol for a prospective pilot study.","volume":"11","author":"Gupta","year":"2022","journal-title":"JMIR Res Protoc"},{"issue":"1","key":"zoi260119r11","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1038\/s41746-023-00979-5","article-title":"Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being.","volume":"6","author":"Li","year":"2023","journal-title":"NPJ Digit Med"},{"key":"zoi260119r13","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2024.102900","article-title":"Transformers and large language models in healthcare: a review.","volume":"154","author":"Nerella","year":"2024","journal-title":"Artif Intell Med"},{"key":"zoi260119r14","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1016\/j.patrec.2023.02.016","article-title":"Transformer-based language models for mental health issues: a survey.","volume":"167","author":"Greco","year":"2023","journal-title":"Pattern Recognit Lett"},{"key":"zoi260119r16","doi-asserted-by":"publisher","first-page":"16","DOI":"10.21037\/jmai-23-136","article-title":"An experimental study of integrating fine-tuned large language models and prompts for enhancing mental health support chatbot system.","volume":"7","author":"Yu","year":"2024","journal-title":"J Med Artif Intell"},{"issue":"1","key":"zoi260119r17","doi-asserted-by":"publisher","first-page":"1177","DOI":"10.1186\/s12913-022-08519-9","article-title":"Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol.","volume":"22","author":"Creed","year":"2022","journal-title":"BMC Health Serv Res"},{"key":"zoi260119r18","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2023.1277756","article-title":"ChatGPT is not ready yet for use in providing mental health assessment and interventions.","volume":"14","author":"Dergaa","year":"2024","journal-title":"Front Psychiatry"},{"issue":"12","key":"zoi260119r23","article-title":"Motivational interviewing for substance use reduction.","volume":"12","author":"Schwenker","year":"2023","journal-title":"Cochrane Database Syst Rev"},{"issue":"8","key":"zoi260119r24","doi-asserted-by":"publisher","first-page":"929","DOI":"10.1007\/s11606-016-3685-3","article-title":"Motivational interviewing improves medication adherence: a systematic review and meta-analysis.","volume":"31","author":"Palacio","year":"2016","journal-title":"J Gen Intern Med"},{"issue":"1","key":"zoi260119r25","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1097\/NHH.0000000000000001","article-title":"Effectiveness of motivational interviewing to improve chronic condition self-management: what does the research show us?","volume":"32","author":"Coyne","year":"2014","journal-title":"Home Healthc Nurse"},{"key":"zoi260119r26","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1016\/j.jsat.2016.01.001","article-title":"The Motivational Interviewing Treatment Integrity Code (MITI 4): rationale, preliminary reliability, and validity.","volume":"65","author":"Moyers","year":"2016","journal-title":"J Subst Abuse Treat"},{"issue":"1","key":"zoi260119r28","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models.","volume":"31","author":"Gallifant","year":"2025","journal-title":"Nat Med"},{"issue":"2","key":"zoi260119r29","doi-asserted-by":"publisher","first-page":"68","DOI":"10.3390\/info15020068","article-title":"Performance of 4 pre-trained sentence transformer models in the semantic query of a systematic review dataset on peri-implantitis.","volume":"15","author":"Galli","year":"2024","journal-title":"Information (Basel)"},{"issue":"4","key":"zoi260119r33","doi-asserted-by":"publisher","first-page":"402","DOI":"10.1176\/appi.focus.20180022","article-title":"The therapeutic alliance: the fundamental element of psychotherapy.","volume":"16","author":"Stubbe","year":"2018","journal-title":"Focus (Am Psychiatr Publ)"},{"key":"zoi260119r15","author":"Wang","year":"2024"},{"key":"zoi260119r22","volume-title":"Motivational Interviewing: Helping People Change","author":"Miller","year":"2012"},{"key":"zoi260119r27","author":"P\u00e9rez-Rosas","year":"2019"},{"key":"zoi260119r2","unstructured":"Phillips L. A closer look at the mental health provider shortage. Counseling Today. May 2023. Accessed June 23, 2025. https:\/\/www.counseling.org\/publications\/counseling-today-magazine\/article-archive\/article\/legacy\/a-closer-look-at-the-mental-health-provider-shortage"},{"key":"zoi260119r12","unstructured":"Vaswani? A, Shazeer? N, Parmar? N, . Attention is all you need.? arXiv. Preprint posted online December 5, 2017. doi:10.48550\/arXiv.1706.03762"},{"key":"zoi260119r19","unstructured":"Chen? L, Preece? DA, Sikka? P, Gross? JJ, Krause? B. A framework for evaluating appropriateness, trustworthiness, and safety in mental wellness AI chatbots.? arXiv. Preprint posted online July 16, 2024. doi:10.48550\/arXiv.2407.11387"},{"key":"zoi260119r20","unstructured":"Zhu? S, Chen? Z, Bi? G, . ?-Arena: interactive assessment and optimization of LLM-based psychological counselors with tripartite feedback.? arXiv. Preprint posted online May 6, 2025. doi:10.48550\/arXiv.2505.03293"},{"key":"zoi260119r21","unstructured":"Marrapese? A, Suleiman? B, Ullah? I, Kim? J. A novel nuanced conversation evaluation framework for large language models in mental health.? arXiv. Preprint posted online March 8, 2024. doi:10.48550\/arXiv.2403.09705"},{"key":"zoi260119r30","unstructured":"Li? J, Li? R, Liu? Q. Beyond static datasets: a deep interaction approach to LLM evaluation.? arXiv. Preprint posted online September 8, 2023. doi:10.48550\/arXiv.2309.04369"},{"key":"zoi260119r31","unstructured":"TherapySimulation. Version 1.0. July 7, 2025. Accessed August 4, 2025. https:\/\/github.com\/teferrabg\/TherapySimulation.git"},{"key":"zoi260119r32","doi-asserted-by":"crossref","unstructured":"Rahimi? H, Mimno? D, Hoover? J, Naacke? H, Constantin? C, Amann? B. Contextualized topic coherence metrics. In: Graham Y, Purver M, eds. Findings of the Association for Computational Linguistics: EACL 2024. Association for Computational Linguistics; 2024:1760-1773. Accessed April 9, 2025. https:\/\/aclanthology.org\/2024.findings-eacl.123\/","DOI":"10.18653\/v1\/2024.findings-eacl.123"},{"key":"zoi260119r34","unstructured":"Kuratov? Y, Bulatov? A, Anokhin? P, . BABILong: testing the limits of LLMs with long context reasoning-in-a-haystack.? arXiv. Preprint posted online June 14, 2024. doi:10.48550\/arXiv.2406.10149"},{"key":"zoi260119r35","unstructured":"Zhang? Y, Das? SSS, Zhang? R. Verbosity ? veracity: demystify verbosity compensation behavior of large language models.? arXiv. Preprint posted online December 7, 2024. doi:10.48550\/arXiv.2411.07858"},{"key":"zoi260119r36","unstructured":"Sutton R. The bitter lesson of machine learning. KDnuggets. Accessed September 4, 2025. https:\/\/www.kdnuggets.com\/the-bitter-lesson-of-machine-learning"},{"key":"zoi260119r37","doi-asserted-by":"crossref","unstructured":"Na? H, Hua? Y, Wang? Z, . 2025. A survey of large language models in psychotherapy: current landscape and future directions. In: Che W, Nabende J, Shutova E, Pilehvar MT, eds. Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics; 2025:7362-7376. Accessed February 13, 2026. https:\/\/aclanthology.org\/2025.findings-acl.385\/","DOI":"10.18653\/v1\/2025.findings-acl.385"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2846839\/teferra_2026_oi_260119_1773672734.38986.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T15:30:32Z","timestamp":1774279832000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2846839"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,23]]},"references-count":37,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3,2]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2026.2750","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,23]]}}}