{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T19:51:29Z","timestamp":1791316289071,"version":"4.3.1"},"reference-count":55,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Family caregivers of children with chronic health conditions experience substantial physical and mental health burdens, including burnout, anxiety, depression, fatigue, and sleep disturbances. Despite this need, validated digital mental health tools tailored to family caregivers remain limited. AI-powered conversational agents offer a promising approach for delivering on-demand, personalized mental health support, yet development and evaluation frameworks for this population are lacking.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This paper describes the iterative development and formative evaluation of COCO (Caring of Caregivers Online), a conversational agent designed for family caregivers of children with chronic health conditions. COCO integrates problem-solving therapy (PST) and motivational interviewing (MI) within a human-in-the-loop development framework that progressed from rule-based interactions to a large language model (LLM)\u2013powered conversational agent.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n COCO was developed across four phases: (1) caregiver persona and dialogue development based on PST and MI; (2) usability testing of a low-fidelity prototype with standardized patients in a single session of PST; (3) usability testing of a high-fidelity prototype with caregivers in a single session of PST (n=38); (4) integration of an LLM into COCO. The Wizard-of-Oz method was used across phases 2 and 3 to collect naturalistic dialogues and refine COCO\u2019s conversational design. In phase 3, usability of COCO was assessed using the System Usability Scale (SUS). Caregiver emotions were measured before and after the session using 6 subscales of the PANAS-X. In phase 4, GPT-4 was integrated into COCO with few-shot learning and evaluated by research team members using the caregiver personas. Descriptive statistics were used to summarize quantitative measures. The MI principles and techniques used by COCO across the 4 phases were coded using the\n Motivational Interviewing Treatment Integrity Coding Manual<\/jats:italic>\n .\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n In phase 1, 4 gold-standard dialogues were developed using caregiver personas. In phase 2, standardized patients described COCO as validating and identified its problem-solving and on-demand support as helpful for caregivers. In phase 3, COCO-Wizard-of-Oz achieved a mean SUS score of 75.6% (SD 12.9%), reflecting acceptable usability. Participants demonstrated significant improvement in negative affect, sadness, guilt, and fatigue following PST sessions (\n P<\/jats:italic>\n <.05). In phase 4, an LLM-powered COCO was developed and demonstrated promising initial conversational capabilities. Across all phases, conversational quality showed progressively improved, with LLM-powered COCO achieving the highest density of MI techniques per turn (2.56) and greater balance across MI strategy types, particularly in seeking collaboration and reflection.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n COCO demonstrated feasibility and usability as a conversational agent for delivering protocolized therapeutic support to family caregivers of children with chronic conditions. The iterative, human-in-the-loop approach supported the development of empathetic and therapeutically grounded responses. More broadly, this study provides a structured framework for systematically integrating and refining evidence-based therapeutic approaches through iterative testing before LLM deployment.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/94175","type":"journal-article","created":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T15:40:12Z","timestamp":1787067612000},"page":"e94175-e94175","source":"Crossref","is-referenced-by-count":0,"title":["Development and Preliminary Evaluation of a Conversational Agent Delivering Problem-Solving Therapy for Family Caregivers of Children With a Chronic Health Condition: Multiphase Mixed Methods Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2712-0948","authenticated-orcid":false,"given":"Weichao","family":"Yuwen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3123-2328","authenticated-orcid":false,"given":"Liying","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1826-7005","authenticated-orcid":false,"given":"Serena Jinchen","family":"Xie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Myra","family":"Divina","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuehong","family":"Fan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2582-2794","authenticated-orcid":false,"given":"William","family":"Kearns","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8651-4066","authenticated-orcid":false,"given":"Teresa M","family":"Ward","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,6]]},"reference":[{"key":"R1","unstructured":"Resendez J Choula RB Cantor K Caldera S Frank L Raimondi A et al. Caregiving in the US 2025. AARP; 2025. URL: https:\/\/www.aarp.org\/content\/dam\/aarp\/ppi\/topics\/ltss\/family-caregiving\/caregiving-in-us-2025.doi.10.26419-2fppi.00373.001.pdf [Accessed 05-09-2026]"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","DOI":"10.3390\/children4050039","article-title":"Predictors of caregiver burden among mothers of children with chronic conditions","volume":"4","author":"Javalkar","journal-title":"Children (Basel)"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","first-page":"251","DOI":"10.5664\/jcsm.4538","article-title":"Sleep patterns, sleep instability, and health related quality of life in parents of ventilator-assisted children","volume":"11","author":"Meltzer","journal-title":"J Clin Sleep Med"},{"issue":"9 Suppl","key":"R4","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1097\/01.NAJ.0000336406.45248.4c","article-title":"Physical and mental health effects of family caregiving","volume":"108","author":"Schulz","journal-title":"Am J Nurs"},{"key":"R5","unstructured":"National Alliance for Caregiving. 2024. URL: https:\/\/www.caregiving.org\/ [Accessed 20-02-2026]"},{"issue":"3","key":"R6","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1016\/j.jpainsymman.2016.03.014","article-title":"Family caregiver depressive symptom and grief outcomes from the ENABLE III randomized controlled trial","volume":"52","author":"Dionne-Odom","journal-title":"J Pain Symptom Manage"},{"issue":"4","key":"R7","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1177\/1043454203254039","article-title":"Psychosocial support for patients in pediatric oncology: the influences of parents, schools, peers, and technology","volume":"20","author":"Suzuki","journal-title":"J Pediatr Oncol Nurs"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1177\/20552076231171967","article-title":"Digital health interventions to support family caregivers: an updated systematic review","volume":"9","author":"Zhai","journal-title":"Digit Health"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1093\/jamiaopen\/ooaa053","article-title":"Harnessing mobile health technology to support long-term chronic illness management: exploring family caregiver support needs in the outpatient setting","volume":"3","author":"Shin","journal-title":"JAMIA Open"},{"issue":"5","key":"R10","doi-asserted-by":"publisher","first-page":"1194","DOI":"10.1055\/a-1967-8721","article-title":"Enhancing resilience in family caregivers using an mHealth app","volume":"13","author":"Smeallie","journal-title":"Appl Clin Inform"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2020.00128","article-title":"Digital mental health tools for caregivers of older adults-a scoping review","volume":"8","author":"Petrovic","journal-title":"Front Public Health"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.17.00110","article-title":"Novel health information technology tool use by adult patients undergoing allogeneic hematopoietic cell transplantation: longitudinal quantitative and qualitative patient-reported outcomes","volume":"2","author":"Runaas","journal-title":"JCO Clin Cancer Inform"},{"issue":"10","key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/14166","article-title":"Conversational agents in the treatment of mental health problems: mixed-method systematic review","volume":"6","author":"Gaffney","journal-title":"JMIR Ment Health"},{"issue":"9","key":"R14","doi-asserted-by":"publisher","first-page":"1248","DOI":"10.1093\/jamia\/ocy072","article-title":"Conversational agents in healthcare: a systematic review","volume":"25","author":"Laranjo","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/365153.365168","article-title":"ELIZA\u2014a computer program for the study of natural language communication between man and machine","volume":"9","author":"Weizenbaum","journal-title":"Commun ACM"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","DOI":"10.2196\/16934","article-title":"Effectiveness of conversational agents (virtual assistants) in health care: protocol for a systematic review","volume":"9","author":"de Cock","journal-title":"JMIR Res Protoc"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1145\/3334480.3382902","article-title":"A wizard-of-oz interface and persona-based methodology for collecting health counseling dialog","author":"Kearns","journal-title":"CHI Conf Hum Factors Comput Syst"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00979-5","article-title":"Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being","volume":"6","author":"Li","journal-title":"NPJ Digit Med"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2022.847991","article-title":"Evaluating the therapeutic alliance with a free-text CBT conversational agent (Wysa): a mixed-methods study","volume":"4","author":"Beatty","journal-title":"Front Digit Health"},{"key":"R20","unstructured":"Chung NC Dyer G Brocki L . Challenges of large language models for mental health counseling. arXiv. Preprint posted online on Nov 23, 2023. [doi: 10.48550\/arXiv.2311.13857]"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Guo T Chen X Wang Y Chang R Pei S Chawla NV et al. Large language model based multi-agents: a survey of progress and challenges. In: Larson K , editor. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. IJCAI Organization; 2024:8048-8057. [doi: 10.24963\/ijcai.2024\/890]","DOI":"10.24963\/ijcai.2024\/890"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/59479","article-title":"The opportunities and risks of large language models in mental health","volume":"11","author":"Lawrence","journal-title":"JMIR Ment Health"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsat.2016.01.001","article-title":"The Motivational Interviewing Treatment Integrity Code (MITI 4): rationale, preliminary reliability and validity","volume":"65","author":"Moyers","journal-title":"J Subst Abuse Treat"},{"issue":"3","key":"R24","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1093\/jamia\/ocv165","article-title":"Self-monitoring practices, attitudes, and needs of individuals with bipolar disorder: implications for the design of technologies to manage mental health","volume":"23","author":"Murnane","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","first-page":"38","DOI":"10.4278\/0890-1171-12.1.38","article-title":"The transtheoretical model of health behavior change","volume":"12","author":"Prochaska","journal-title":"Am J Health Promot"},{"key":"R26","unstructured":"Wang L Mujib MI Williams J Demiris G Huh-Yoo J . An evaluation of generative pre-training model-based therapy chatbot for caregivers. arXiv. Preprint posted online on Jul 28, 2021. [doi: 10.48550\/arXiv.2107.13115]"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.mex.2025.103272","article-title":"Effectiveness of Chatbot interventions for reducing caregiver burden: protocol for a systematic review and meta-analysis","volume":"14","author":"Shankar","journal-title":"MethodsX"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.2196\/63715","article-title":"The PDC30 chatbot-development of a psychoeducational resource on dementia caregiving among family caregivers: mixed methods acceptability study","volume":"8","author":"Cheng","journal-title":"JMIR Aging"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1007\/s12160-016-9830-8","article-title":"Just-in-time adaptive interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support","volume":"52","author":"Nahum-Shani","journal-title":"Ann Behav Med"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.cpr.2005.12.005","article-title":"The efficacy of problem solving therapy in reducing mental and physical health problems: a meta-analysis","volume":"27","author":"Malouff","journal-title":"Clin Psychol Rev"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.1138","article-title":"Periodic prompts and reminders in health promotion and health behavior interventions: systematic review","volume":"11","author":"Fry","journal-title":"J Med Internet Res"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1177\/0890117116671082","article-title":"Tailored educational approaches for consumer health: a model to address health promotion in an era of personalized medicine","volume":"32","author":"Cohn","journal-title":"Am J Health Promot"},{"issue":"6","key":"R33","doi-asserted-by":"publisher","first-page":"648","DOI":"10.1080\/08839514.2010.492259","article-title":"Maintaining engagement in long-term interventions with relational agents","volume":"24","author":"Bickmore","journal-title":"Appl Artif Intell"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.2196\/24172","article-title":"Personas for better targeted eHealth technologies: user-centered design approach","volume":"9","author":"Ten Klooster","journal-title":"JMIR Hum Factors"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1097\/NCN.0b013e31819f7c7c","article-title":"User-centered design and interactive health technologies for patients","volume":"27","author":"De Vito Dabbs","journal-title":"Comput Inform Nurs"},{"issue":"12","key":"R36","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1016\/j.puhe.2012.08.007","article-title":"Depression among caregivers of children with asthma and its impact on communication with health care providers","volume":"126","author":"Fagnano","journal-title":"Public Health"},{"issue":"7","key":"R37","doi-asserted-by":"publisher","first-page":"731","DOI":"10.1080\/02770900903082571","article-title":"Barriers to asthma management among urban families: caregiver and child perspectives","volume":"46","author":"Laster","journal-title":"J Asthma"},{"issue":"6","key":"R38","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1093\/jpepsy\/jsw007","article-title":"Daily sleep patterns, sleep quality, and sleep hygiene among parent-child dyads of young children newly diagnosed with juvenile idiopathic arthritis and typically developing children","volume":"41","author":"Yuwen","journal-title":"J Pediatr Psychol"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1016\/j.pedn.2017.07.007","article-title":"Struggling in the dark to help my child: parents\u2019 experience in caring for a young child with juvenile idiopathic arthritis","volume":"37","author":"Yuwen","journal-title":"J Pediatr Nurs"},{"issue":"10","key":"R40","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.71224","article-title":"The evolution from standardized to virtual patients in medical education","volume":"16","author":"Hamilton","journal-title":"Cureus"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1145\/357417.357420","article-title":"An iterative design methodology for user-friendly natural language office information applications","volume":"2","author":"Kelley","journal-title":"ACM Trans Inf Syst"},{"key":"R42","unstructured":"Joglekar C . WOzBot: a wizard of oz based method for chatbot response improvement [Master\u2019s Thesis]. Trinity College Dublin, University of Dublin; 2022. URL: https:\/\/publications.scss.tcd.ie\/theses\/diss\/2022\/TCD-SCSS-DISSERTATION-2022-127.pdf [Accessed 05-09-2026]"},{"key":"R43","doi-asserted-by":"crossref","unstructured":"Watson D Clark LA . The PANAS-X: manual for the positive and negative affect schedule - expanded form. The University of Iowa; 1994. URL: https:\/\/iro.uiowa.edu\/esploro\/fulltext\/other\/The-PANAS-X-Manual-for-the-Positive\/9983557488402771?repId=12674991360002771&mId=13675073940002771&institution=01IOWA_INST [Accessed 05-09-2026]","DOI":"10.17077\/48vt-m4t2"},{"issue":"1","key":"R44","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1080\/10447319509526110","article-title":"IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use","volume":"7","author":"Lewis","journal-title":"Int J Hum Comput Interact"},{"issue":"3","key":"R45","first-page":"114","volume":"4","author":"Bangor","journal-title":"J Usability Stud"},{"key":"R46","volume":"2024","author":"Filienko","journal-title":"AMIA Annu Symp Proc"},{"key":"R47","unstructured":"Sauro J . Measuring usability with the System Usability Scale (SUS). MeasuringU. Mar 19, 2011. URL: https:\/\/measuringu.com\/sus\/ [Accessed 05-09-2026]"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1016\/j.jretconser.2023.103666","article-title":"Rapport with a chatbot? The underlying role of anthropomorphism in socio-cognitive perceptions of rapport and e-word of mouth","volume":"77","author":"Kayeser Fatima","journal-title":"J Retail Consum Serv"},{"issue":"7","key":"R49","doi-asserted-by":"publisher","first-page":"1620","DOI":"10.1080\/10447318.2022.2146227","article-title":"Influence of rapport and social presence with an AI psychotherapy chatbot on users\u2019 self-disclosure","volume":"40","author":"Lee","journal-title":"Int J Hum Comput Interact"},{"issue":"5","key":"R50","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1177\/0956797612462222","article-title":"The wear and tear of daily stressors on mental health","volume":"24","author":"Charles","journal-title":"Psychol Sci"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2026.1812638","article-title":"Mapping the evidence base of positive psychology interventions: effectiveness, limitations, and future directions","volume":"17","author":"Akhter","journal-title":"Front Psychol"},{"issue":"4","key":"R52","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2024.e26297","article-title":"Ethical and regulatory challenges of AI technologies in healthcare: a narrative review","volume":"10","author":"Mennella","journal-title":"Heliyon"},{"key":"R53","doi-asserted-by":"crossref","unstructured":"AlMakinah R Norcini-Pala A Disney L Canbaz MA . Enhancing mental health support through human-AI collaboration: toward secure and empathetic AI-enabled chatbots. In: 2025 IEEE Conference on Artificial Intelligence (CAI). IEEE; 2026:196-202. [doi: 10.1109\/CAI64502.2025.00038]","DOI":"10.1109\/CAI64502.2025.00038"},{"issue":"10","key":"R54","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare14101395","article-title":"Safety mechanisms and risk mitigation in generative AI mental health chatbots: a systematic scoping review","volume":"14","author":"Olisaeloka","journal-title":"Healthcare (Basel)"},{"key":"R55","unstructured":"Hong S Zhuge M Chen J Zheng X Cheng Y Zhang C et al. MetaGPT: meta programming for a multi-agent collaborative framework. arXiv. Preprint posted online on Aug 1, 2023. [doi: 10.48550\/arXiv.2308.00352]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T19:11:57Z","timestamp":1791313917000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e94175"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,6]]},"references-count":55,"URL":"https:\/\/doi.org\/10.2196\/94175","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,6]]},"article-number":"v28i5e94175"}}