{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:06Z","timestamp":1785544986754,"version":"3.56.0"},"reference-count":23,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Therapeutic chatbots are increasingly deployed across digital mental health services, yet most evaluation efforts remain diagnostic rather than actionable. Organizations lack structured pathways to translate evaluation findings into validated quality improvements aligned with health care quality assurance requirements.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to introduce EvaluationPlus, a decision support framework that operationalizes a reproducible evaluation-to-enhancement loop for therapeutic artificial intelligence systems. We aimed to demonstrate its feasibility through expert-guided diagnosis, multi-large language model (LLM) enhancement mapping, and within-subject validation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Using the bilingual mental health chatbot Dr. CareSam (GPT 4.0\u2013based), we conducted 3 iterative enhancement cycles. Two licensed clinical psychologists performed structured diagnostic reviews using think-aloud protocols to identify competency-specific deficits across a 7-dimension therapeutic competency rubric. Three LLMs (GPT 4.0, Claude 4.0 Sonnet, Gemini 2.5 Flash) generated prescriptive enhancement strategies aligned with identified gaps. A participant-blinded, within-subject A\/B validation study with Korean graduate students from the Department of Applied Artificial Intelligence, Sungkyunkwan University (N=15; 16 recruited, 1 excluded; IRB-approved) compared baseline and enhanced versions across standardized clinical scenarios spanning mild anxiety, to crisis-level presentations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The enhanced system demonstrated substantial improvement in overall therapeutic quality, with mean scores increasing from 5.40 to 7.63 (\u0394 =+2.23 points, 41%; dz=0.881; 95% bootstrap CI [0.32-2.20]). Prespecified target dimensions \u2014 active listening and appropriate questions, personalization, and complex thinking \u2014 showed large-effect improvements (mean gain +3.04; dz range 0.96\u20101.08), significantly exceeding gains in nontargeted dimensions (+1.62; targeting differential +1.42 points). Directional improvement was observed in 13 of 15 participants (86.7%). User preference strongly favored the enhanced system (13\/15, 86.7%), and expert clinical evaluation confirmed maintained safety and therapeutic appropriateness across four scenario severity levels (preference rate 75%; 3 of 4 scenarios). Cross-participant rating consistency improved substantially (coefficient of variation: 20% \u2192 8.1%).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>EvaluationPlus demonstrates feasibility as a structured framework for iterative quality assurance of therapeutic artificial intelligence systems. By linking expert diagnostic procedures with prescriptive multi-LLM enhancement mapping and multistakeholder validation, the framework supports reproducible improvement cycles relevant to organizational oversight of digital mental health tools. Limitations include a small pilot sample, single-culture focus, and simulated crisis scenarios; future work should extend validation to diverse clinical populations and longitudinal outcome assessment.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/87887","type":"journal-article","created":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T20:00:09Z","timestamp":1781553609000},"page":"e87887-e87887","source":"Crossref","is-referenced-by-count":0,"title":["Decision Support Framework for Quality Assurance and Enhancement of Therapeutic Artificial Intelligence Systems: Mixed Methods Pilot Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-4338-8728","authenticated-orcid":false,"given":"Boyoung","family":"Kang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3082-5172","authenticated-orcid":false,"given":"Kyungmin","family":"Kwon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6986-8280","authenticated-orcid":false,"given":"Piao","family":"Huilin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3780-0603","authenticated-orcid":false,"given":"Seohyeon","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3913-0940","authenticated-orcid":false,"given":"Seoin","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7362-5138","authenticated-orcid":false,"given":"Hayoung","family":"Oh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,23]]},"reference":[{"issue":"14","key":"R1","doi-asserted-by":"publisher","first-page":"2955","DOI":"10.1017\/S0033291716001665","article-title":"Mental disorders among college students in the World Health Organization world mental health surveys","volume":"46","author":"Auerbach","journal-title":"Psychol Med"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","DOI":"10.1002\/mpr.1759","article-title":"Internet interventions for mental health in university students: a systematic review and meta-analysis","volume":"28","author":"Harrer","journal-title":"Int J Methods Psychiatr Res"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1176\/appi.ps.201800332","article-title":"Increased rates of mental health service utilization by U.S. college students: 10-year population-level trends (2007-2017)","volume":"70","author":"Lipson","journal-title":"Psychiatr Serv"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"National Survey on Mental Health. Ministry of Health and Welfare; 2021. [doi: 10.30773\/pi.2022.0307]","DOI":"10.30773\/pi.2022.0307"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"National Center for Health Statistics. Health, United States, 2020\u20132021: Annual Perspective. NCHS; 2023. [doi: 10.15620\/cdc:122044]","DOI":"10.15620\/cdc:122044"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/63538","article-title":"Development and evaluation of a mental health chatbot using ChatGPT 4.0: mixed methods user experience study with Korean users","volume":"13","author":"Kang","journal-title":"JMIR Med Inform"},{"key":"R7","doi-asserted-by":"crossref","unstructured":"Chu SY Kim JW Yi MY . Think together and work better: combining humans\u2019 and llms\u2019 think-aloud outcomes for effective text evaluation. Presented at: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI \u201925). Apr 26 to May 1, 2025; ACM. Yokohama Japan. Apr 26, 2025.[doi: 10.1145\/3706598.3713181]","DOI":"10.1145\/3706598.3713181"},{"issue":"9","key":"R8","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","journal-title":"Nat Med"},{"key":"R9","unstructured":"Park JI Abbasian M Azimi I Bounds DT Jun A Han J et al. Building trust in mental health chatbots: safety metrics and LLM-based evaluation tools. arXiv. Preprint posted online on Aug 3, 2024. [doi: 10.48550\/arXiv.2408.04650]"},{"key":"R10","doi-asserted-by":"crossref","unstructured":"Ericsson KA Simon HA . Protocol Analysis: Verbal Reports as Data. Rev ed ed. MIT Press; 1993. [doi: 10.7551\/mitpress\/5657.001.0001] ISBN: 9780262272391","DOI":"10.7551\/mitpress\/5657.001.0001"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/63058","article-title":"Advancing clinical chatbot validation using AI-powered evaluation with a new 3-bot evaluation system: instrument validation study","volume":"8","author":"Choo","journal-title":"JMIR Nurs"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"key":"#cr-split#-R13.1","doi-asserted-by":"crossref","unstructured":"Louie R Nandi A Fang W Chang C Brunskill E Yang D . Roleplay-doh: enabling domain-experts to create LLM-simulated patients via eliciting and adhering to principles. Presented at: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). Nov 12-16, 2024:ACL. 10570-10603","DOI":"10.18653\/v1\/2024.emnlp-main.591"},{"key":"#cr-split#-R13.2","doi-asserted-by":"crossref","unstructured":"Miami, Florida, USA. 2024.[doi: 10.18653\/v1\/2024.emnlp-main.591]","DOI":"10.18653\/v1\/2024.emnlp-main.591"},{"key":"#cr-split#-R14.1","doi-asserted-by":"crossref","unstructured":"Moilanen J Visuri A Suryanarayana SA Alorwu A Yatani K Hosio S . Measuring the effect of mental health chatbot personality on user engagement. Presented at: MUM 2022. Nov 27-30, 2022:ACM. 138-150","DOI":"10.1145\/3568444.3568464"},{"key":"#cr-split#-R14.2","doi-asserted-by":"crossref","unstructured":"Lisbon Portugal. Nov 27, 2022.[doi: 10.1145\/3568444.3568464]","DOI":"10.1145\/3568444.3568464"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Wei J Wang X Schuurmans D Bosma M Ichter B Xia F et al. Chain-of-thought prompting elicits reasoning in large language models. In: Adv Neural Inf Process Syst. Curran Associates; 2022. [doi: 10.5555\/3600270.3602070]","DOI":"10.52202\/068431-1800"},{"key":"R16","unstructured":"Kim T Bae S Kim HA Lee SW Hong H Yang C et al. Evaluating and auditing LLM-driven chatbots for psychiatric patients in clinical mental health settings. Presented at: CHI 2024 Workshop on Human-centered Evaluation and Auditing of Language Models (HEAL); May 12, 2024; Honolulu, HI. URL: https:\/\/younghokim.net\/files\/papers\/mindfuldiary-kim-heal24.pdf [Accessed 10-07-2026]"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01684-1","article-title":"Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis","volume":"8","author":"Gaber","journal-title":"NPJ Digit Med"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Association AP . Diagnostic and Statistical Manual of Mental Disorders. 5th ed ed. American Psychiatric Publishing; 2013. [doi: 10.1176\/appi.books.9780890425596]","DOI":"10.1176\/appi.books.9780890425596"},{"key":"R19","unstructured":"Hill CE . Helping Skills: Facilitating Exploration, Insight, and Action. 3rd ed ed. American Psychological Association; 2009. ISBN: 9781433804519"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1080\/10904018.2013.813234","article-title":"The relative effectiveness of active listening in initial interactions","volume":"28","author":"Weger","journal-title":"Int J Listen"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Norcross JC Lambert MJ . Psychotherapy Relationships That Work: Volume 1 \u2014 Evidence-Based Therapist Contributions. 3rd ed. Oxford University Press; 2019. [doi: 10.1093\/med-psych\/9780190843953.003.0001] ISBN: 9780190843953","DOI":"10.1093\/med-psych\/9780190843953.003.0001"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e87887","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:06Z","timestamp":1785536706000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e87887"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,23]]},"references-count":23,"URL":"https:\/\/doi.org\/10.2196\/87887","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,23]]},"article-number":"v14i14e87887"}}