{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T20:12:45Z","timestamp":1788293565144,"version":"build-2803163510"},"reference-count":51,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Ment Health"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Mental health chatbots are increasingly used to support people with depressive symptoms, and large language models make these systems more flexible than rule-based chatbots. However, it remains unclear how well large language model\u2013based chatbots deliver structured psychological interventions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study examined how well a GPT-4o\u2013based chatbot delivered a behavioral activation intervention for young people with depression using sessions with artificial users and clinical expert assessment. It also identified limitations and potential refinements.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We implemented a GPT-4o (gpt-4o-2024-08-06; OpenAI)\u2013based chatbot using a structured system prompt to deliver a single-session behavioral activation intervention for people with depression aged 14 to 29 years. We generated 48 sessions with GPT-4o\u2013based artificial users derived from clinical vignettes varying across 7 characteristics. Ten clinical experts, either licensed psychotherapists or advanced psychotherapy trainees, independently assessed the sessions using the 14-item Quality of Behavioral Activation Scale (Q-BAS), rated from 0 to 6, supplemented by rating therapeutic capabilities, artificial user authenticity and difficulty, and qualitative feedback.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The chatbot completed all 7 intervention phases in every session. The mean holistic session quality rating was 3.94 (SD 1.23), and the mean Q-BAS rating was 4.03 (SD 1.18). Thirteen of 14 Q-BAS components exceeded the satisfactory threshold of 3. Ratings were highest for mood assessment (mean 5.42, SD 1.09) and activity planning (mean 4.98, SD 1.41) and lowest for explaining positive reinforcement (mean 2.92, SD 2.30) and supporting activity-mood monitoring (mean 3.02, SD 2.04). Therapeutic capability ratings were highest for message safety (mean 5.90, SD 0.37), message clarity (mean 5.56, SD 0.77), and objective, nonjudgmental communication (mean 5.17, SD 1.04) and lowest for therapeutic rapport (mean 4.12, SD 1.45) and natural conversation flow (mean 4.25, SD 1.42). Artificial users were rated below the scale midpoint for authenticity (mean 2.75, SD 1.41) and difficulty (mean 1.23, SD 1.46). Clinical experts described the chatbot as structured, clear, and safe but identified insufficient clinical reasoning as the main limitation, particularly in evaluating the therapeutic suitability and feasibility of activities, barriers, solution strategies, and rewards. Artificial users were often highly compliant, especially when identifying positive activities.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>In expert-rated sessions with artificial users, the chatbot delivered the behavioral activation intervention as intended and performed strongest on procedural components. It performed less well on positive reinforcement and activity-mood monitoring, indicating refinement needs in clinical reasoning, follow-up questioning, and evaluating whether proposed activities, plans, barriers, solution strategies, and rewards are therapeutically appropriate and feasible. The findings identify targets for improvement before testing with human users, while the artificial user design and expert ratings limit conclusions about real therapeutic interactions.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/94781","type":"journal-article","created":{"date-parts":[[2026,8,12]],"date-time":"2026-08-12T15:00:05Z","timestamp":1786546805000},"page":"e94781-e94781","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Model\u2013Based Behavioral Activation Chatbot for Young People With Depression Using Artificial Users and Clinical Experts: Mixed Methods Evaluation"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1032-6982","authenticated-orcid":false,"given":"Florian Onur","family":"Kuhlmeier","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2002-0725","authenticated-orcid":false,"given":"Leon","family":"Hanschmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5602-2974","authenticated-orcid":false,"given":"Melina","family":"Rabe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9194-276X","authenticated-orcid":false,"given":"Stefan","family":"L\u00fcttke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9589-3697","authenticated-orcid":false,"given":"Eva-Lotta","family":"Brakemeier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6546-4816","authenticated-orcid":false,"given":"Alexander","family":"Maedche","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,1]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1002\/wps.21299","article-title":"The evolving field of digital mental health: current evidence and implementation issues for smartphone apps, generative artificial intelligence, and virtual reality","volume":"24","author":"Torous","journal-title":"World Psychiatry"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785","article-title":"Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial","volume":"4","author":"Fitzpatrick","journal-title":"JMIR Ment Health"},{"issue":"11","key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/12106","article-title":"An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study","volume":"6","author":"Inkster","journal-title":"JMIR mHealth uHealth"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/28003","article-title":"The challenges in designing a prevention chatbot for eating disorders: observational study","volume":"6","author":"Chan","journal-title":"JMIR Form Res"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1002\/wps.21183","article-title":"Current evidence on the efficacy of mental health smartphone apps for symptoms of depression and anxiety. a meta-analysis of 176 randomized controlled trials","volume":"23","author":"Linardon","journal-title":"World Psychiatry"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/44838","article-title":"An overview of chatbot-based mobile mental health apps: insights from app description and user reviews","volume":"11","author":"Haque","journal-title":"JMIR mHealth uHealth"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1136\/bmjment-2023-300884","article-title":"ChatGPT and mental healthcare: balancing benefits with risks of harms","volume":"26","author":"Blease","journal-title":"BMJ Ment Health"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1038\/s44184-024-00056-z","article-title":"Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation","volume":"3","author":"Stade","journal-title":"Npj Ment Health Res"},{"issue":"9","key":"R9","doi-asserted-by":"publisher","first-page":"1192","DOI":"10.1080\/13607863.2018.1484888","article-title":"Treatment fidelity evidence for BE-ACTIV - a behavioral intervention for depression in nursing homes","volume":"23","author":"Meeks","journal-title":"Aging Ment Health"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/57400","article-title":"Large language models for mental health applications: systematic review","volume":"11","author":"Guo","journal-title":"JMIR Ment Health"},{"issue":"8","key":"R11","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmen.0000426","article-title":"Correction: when ELIZA meets therapists: a Turing test for the heart and mind","volume":"2","author":"Hatch","journal-title":"PLOS Ment Health"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2400802","article-title":"Randomized trial of a generative AI chatbot for mental health treatment","volume":"2","author":"Heinz","journal-title":"NEJM AI"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01611-4","article-title":"A scoping review of large language models for generative tasks in mental health care","volume":"8","author":"Hua","journal-title":"NPJ Digit Med"},{"issue":"5","key":"R14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3398069","article-title":"Machine learning in mental health: a systematic review of the HCI literature to support the development of effective and implementable ML systems","volume":"27","author":"Thieme","journal-title":"ACM Trans Comput-Hum Interact"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/90759","article-title":"Toward retrieval-grounded evaluation for conversational large language model-based risk assessment","volume":"5","author":"Hu","journal-title":"JMIR AI"},{"issue":"7","key":"R16","doi-asserted-by":"publisher","first-page":"886","DOI":"10.1080\/10503307.2023.2197630","article-title":"Individual behavioral activation in the treatment of depression: a meta analysis","volume":"33","author":"Cuijpers","journal-title":"Psychother Res"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1145\/3613905.3650860","article-title":"Generating personas using LLMs and assessing their viability","author":"Schuller","journal-title":"Extended Abstracts CHI Conf Hum Factors Comput Syst"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1145\/3706598.3714014","article-title":"Scaffolding empathy: training counselors with simulated patients and utterance-level performance visualizations","author":"Steenstra","journal-title":"Proc 2025 CHI Conf Hum Factors Comput Syst"},{"key":"R19","unstructured":"Qiu H Lan Z . Interactive agents: simulating counselor-client psychological counseling via role-playing LLM-to-LLM interactions. arXiv. Preprint posted online on Aug 28, 2024. [doi: 10.48550\/arXiv.2408.15787]"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.711","article-title":"PATIENT-\ud835\udf13: using large language models to simulate patients for training mental health professionals","author":"Wang","journal-title":"Proc 2024 Conf Empir Methods Nat Lang Process"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.591","article-title":"Roleplay-doh: enabling domain-experts to create LLM-simulated patients via eliciting and adhering to principles","author":"Louie","journal-title":"Proc 2024 Conf Empir Methods Nat Lang Process"},{"issue":"2","key":"R22","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/j.jaac.2016.11.007","article-title":"Little treatments, promising effects? Meta-analysis of single-session interventions for youth psychiatric problems","volume":"56","author":"Schleider","journal-title":"J Am Acad Child Adolesc Psychiatry"},{"issue":"2","key":"R23","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1038\/s41562-021-01235-0","article-title":"A randomized trial of online single-session interventions for adolescent depression during COVID-19","volume":"6","author":"Schleider","journal-title":"Nat Hum Behav"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/66632","article-title":"Designing chatbots to treat depression in youth: qualitative study","volume":"12","author":"Kuhlmeier","journal-title":"JMIR Hum Factors"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Kuhlmeier FO Gnewuch U L\u00fcttke S Brakemeier EL M\u00e4dche A . A personalized conversational agent to treat depression in youth and young adults \u2013 a transdisciplinary design science research project. In: Drechsler A Gerber A Hevner A , editors. The Transdisciplinary Reach of Design Science Research: 17th International Conference on Design Science Research in Information Systems and Technology, DESRIST 2022, St Petersburg, FL, USA, June 1\u20133, 2022, Proceedings. Springer International Publishing; 2022:30-41. [doi: 10.1007\/978-3-031-06516-3_3]","DOI":"10.1007\/978-3-031-06516-3_3"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajp.2023.103770","article-title":"Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettes","volume":"89","author":"Franco D\u2019Souza","journal-title":"Asian J Psychiatr"},{"issue":"3","key":"R27","doi-asserted-by":"publisher","DOI":"10.2196\/24387","article-title":"Barriers to and facilitators of user engagement with digital mental health interventions: systematic review","volume":"23","author":"Borghouts","journal-title":"J Med Internet Res"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2018.09.015","article-title":"Adolescent and adult differences in major depression symptom profiles","volume":"243","author":"Rice","journal-title":"J Affect Disord"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3635472","article-title":"Between rhetoric and reality: real-world barriers to uptake and early engagement in digital mental health interventions","volume":"31","author":"Jardine","journal-title":"ACM Trans Comput-Hum Interact"},{"key":"R30","unstructured":"Gnewuch U Yu M Maedche A . The effect of perceived similarity in dominance on customer self-disclosure to chatbots in conversational commerce. Presented at: Proceedings of the 28th European Conference on Information Systems (ECIS); Jun 15-17, 2020; Marrakech, Morocco. URL: https:\/\/aisel.aisnet.org\/ecis2020_rp\/53 [Accessed 20-08-2026]"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1037\/ccp0000151","article-title":"A pragmatic randomized clinical trial of behavioral activation for depressed pregnant women","volume":"85","author":"Dimidjian","journal-title":"J Consult Clin Psychol"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","DOI":"10.1186\/s40359-024-02053-5","article-title":"Pilot evaluation on an adapted tele-behavioral activation to increase physical activity in persons with depression: a single-arm pilot study","volume":"12","author":"Rethorst","journal-title":"BMC Psychol"},{"issue":"4","key":"R33","doi-asserted-by":"publisher","first-page":"731","DOI":"10.1007\/s41347-025-00519-w","article-title":"A framework for evaluating mental health artificial intelligence-based conversational agents","volume":"10","author":"Bunge","journal-title":"J technol behav sci"},{"key":"R34","doi-asserted-by":"crossref","unstructured":"Mayring P Fenzl T . Qualitative inhaltsanalyse. In: Baur N Blasius J , editors. Handbuch Methoden Der Empirischen Sozialforschung [Book in German]. 2nd ed. Springer Fachmedien Wiesbaden; 2019:633-648. [doi: 10.1007\/978-3-658-21308-4_42]","DOI":"10.1007\/978-3-658-21308-4_42"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642761","article-title":"Facilitating self-guided mental health interventions through human-language model interaction: a case study of cognitive restructuring","author":"Sharma","journal-title":"Proc ACM Conf Hum Factors Comput Syst"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2025.103514","article-title":"LLM-based conversational agents for behaviour change support: a randomised controlled trial examining efficacy, safety, and the role of user behaviour","volume":"200","author":"Meyer","journal-title":"Int J Hum Comput Stud"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/67365","article-title":"Safety and user experience of a generative artificial intelligence digital mental health intervention: exploratory randomized controlled trial","volume":"27","author":"Campellone","journal-title":"J Med Internet Res"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.2196\/70014","article-title":"Evaluating generative AI in mental health: systematic review of capabilities and limitations","volume":"12","author":"Wang","journal-title":"JMIR Ment Health"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-64769-1","article-title":"Quantifying the reasoning abilities of LLMs on clinical cases","volume":"16","author":"Qiu","journal-title":"Nat Commun"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1609\/aies.v7i1.31654","article-title":"Risks from language models for automated mental healthcare: ethics and structure for implementation (extended abstract)","volume":"7","author":"Grabb","journal-title":"Proc AAAI ACM Conf AI Ethics Soc"},{"key":"R41","doi-asserted-by":"crossref","unstructured":"Xie C Zhu D Wang Z Zhang H Wei Z . Explainable AI for mental health detection from social media: a comparative study of traditional machine learning and a large language model. SSRN. Preprint posted online on Mar 14, 2026. [doi: 10.2139\/ssrn.6429778]","DOI":"10.2139\/ssrn.6429778"},{"issue":"5","key":"R42","doi-asserted-by":"publisher","DOI":"10.2196\/27868","article-title":"Evidence of human-level bonds established with a digital conversational agent: cross-sectional, retrospective observational study","volume":"5","author":"Darcy","journal-title":"JMIR Form Res"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.2196\/56529","article-title":"Considering the role of human empathy in AI-driven therapy","volume":"11","author":"Rubin","journal-title":"JMIR Ment Health"},{"issue":"3","key":"R44","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1093\/jamia\/ocad222","article-title":"Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review","volume":"31","author":"Ding","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"R45","doi-asserted-by":"publisher","DOI":"10.2196\/15823","article-title":"Responses of conversational agents to health and lifestyle prompts: investigation of appropriateness and presentation structures","volume":"22","author":"Kocaballi","journal-title":"J Med Internet Res"},{"key":"R46","unstructured":"Gao L Sherwood J Aleisa N Damoah A Lu Y Qu X . Human-centered AI agents for healthcare and education: a systematic literature review. Presented at: Human-Computer Interaction International (HCII); Jun 22-27, 2025; Gothenburg, Sweden. URL: https:\/\/faculty.cs.gwu.edu\/xiaodongqu\/papers\/HCII_2025_5774_AI_Agent.pdf [Accessed 30-06-2026]"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.1145\/3706598.3713220","article-title":"Simulacrum of stories: examining large language models as qualitative research participants","author":"Kapania","journal-title":"Proc 2025 CHI Conf Hum Factors Comput Syst"},{"key":"R48","unstructured":"Ganguli D Lovitt L Kernion J et al. Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. arXiv. Preprint posted online on Aug 23, 2022. [doi: 10.48550\/arXiv.2209.07858]"},{"issue":"2","key":"R49","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","journal-title":"J Chiropr Med"},{"issue":"1","key":"R50","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1037\/1082-989X.1.1.30","article-title":"Forming inferences about some intraclass correlation coefficients","volume":"1","author":"McGraw","journal-title":"Psychol Methods"},{"issue":"1","key":"R51","doi-asserted-by":"publisher","DOI":"10.1525\/collabra.33267","article-title":"Sample size justification","volume":"8","author":"Lakens","journal-title":"Collabra Psychol"}],"container-title":["JMIR Mental Health"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T19:13:02Z","timestamp":1788289982000},"score":1,"resource":{"primary":{"URL":"https:\/\/mental.jmir.org\/2026\/1\/e94781"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,1]]},"references-count":51,"URL":"https:\/\/doi.org\/10.2196\/94781","relation":{},"ISSN":["2368-7959"],"issn-type":[{"value":"2368-7959","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,1]]},"article-number":"v13i2e94781"}}