{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:54Z","timestamp":1785545034110,"version":"3.56.0"},"reference-count":57,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n With the expanding need for advance care planning (ACP), innovative educational strategies for training health care providers are increasingly required. Large language model (LLM)\u2013based ACP chatbots offer a novel and potentially effective solution to enhance health care providers\u2019 competence in navigating complex ACP conversations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop a Chinese-context ACP corpus to support an LLM-based chatbot and evaluate the feasibility and performance of a multi-agent system for simulating complex ACP discussions as a training tool for health care providers.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This study involved dataset construction and model adaptation and evaluation. We constructed 3 structured datasets using synthetic dialogue data generated through prompts derived from ACP-related scientific literature and policy documents. Both open-source (Zhongjing) and closed-source LLMs (GPT-4o-mini) were chosen as baseline models. The Zhongjing model was adapted through fine-tuning, whereas GPT-4o-mini was adapted using both fine-tuning and prompt engineering. Model performance was assessed through automatic and human evaluations following the QUEST (Quality of information, Understanding and reasoning, Expression style and persona, Safety and harm, and Trust and confidence) framework. Statistical comparisons between baseline and adapted models were performed using repeated-measures ANOVA.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Three separate datasets for the assistant, vignette, and evaluator agents were created, which collectively formed a multi-agent artificial intelligence system for Chinese ACP training. The assistant dataset included 4364 dialogues, the vignette dataset comprised 671 clinical scenarios, and the evaluator dataset contained 671 records. Both automatic and human evaluations confirmed that the adapted models significantly outperformed baseline models on most aspects of Chinese ACP conversations and summarization (\u03b7\n 2<\/jats:sup>\n p<\/jats:sub>\n =0.12\u20100.99;\n P<\/jats:italic>\n values ranged from .03 to <.001).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study demonstrates the adequate technical feasibility of the multi-agent LLM-based system for ACP training among health care providers in the Chinese context. Despite its potential as a supportive educational tool, further validation in real-world training contexts is required to establish its effectiveness in enhancing health care providers\u2019 ACP competencies.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/87288","type":"journal-article","created":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T18:05:40Z","timestamp":1785261940000},"page":"e87288-e87288","source":"Crossref","is-referenced-by-count":0,"title":["A Large Language Model\u2013Driven System for Advance Care Planning Training Among Health Care Providers in the Chinese Context: Development and Technical Evaluation"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4341-1892","authenticated-orcid":false,"given":"Minghui","family":"Tan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9940-5072","authenticated-orcid":false,"given":"Siyuan","family":"Tang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0097-6196","authenticated-orcid":false,"given":"Shichao","family":"Kan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6891-244X","authenticated-orcid":false,"given":"Bei","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9185-9894","authenticated-orcid":false,"given":"Zhao","family":"Ni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3111-9559","authenticated-orcid":false,"given":"Haojie","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8783-8919","authenticated-orcid":false,"given":"Jinfeng","family":"Ding","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,28]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.f6064","article-title":"An introduction to advance care planning in practice","volume":"347","author":"Mullick","journal-title":"BMJ"},{"issue":"9","key":"R2","doi-asserted-by":"publisher","first-page":"e543","DOI":"10.1016\/S1470-2045(17)30582-X","article-title":"Definition and recommendations for advance care planning: an international consensus supported by the European Association for Palliative Care","volume":"18","author":"Rietjens","journal-title":"Lancet Oncol"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"234","DOI":"10.1111\/jgs.16801","article-title":"Deconstructing the complexities of advance care planning outcomes: what do we know and where do we go? A scoping review","volume":"69","author":"McMahan","journal-title":"J Am Geriatr Soc"},{"issue":"7","key":"R4","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2021-060201","article-title":"What is the evidence for efficacy of advance care planning in improving patient outcomes? A systematic review of randomised controlled trials","volume":"12","author":"Malhotra","journal-title":"BMJ Open"},{"issue":"e3","key":"R5","doi-asserted-by":"publisher","first-page":"e537","DOI":"10.1136\/spcare-2021-003488","article-title":"Advance care planning and caregiver outcomes: intervention efficacy \u2013 systematic review","volume":"13","author":"Malhotra","journal-title":"BMJ Support Palliat Care"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1136\/bmjspcare-2018-001549","article-title":"General practice palliative care: patient and carer expectations, advance care plans and place of death\u2014a systematic review","volume":"14","author":"Johnson","journal-title":"BMJ Support Palliat Care"},{"issue":"23-24","key":"R7","doi-asserted-by":"publisher","first-page":"4276","DOI":"10.1111\/jocn.15049","article-title":"Barriers to advance care planning with patients as perceived by nurses and other healthcare professionals: a systematic review","volume":"28","author":"Blackwood","journal-title":"J Clin Nurs"},{"issue":"2","key":"R8","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1093\/gerona\/glab091","article-title":"\u201cDon\u2019t talk to them about goals of care\u201d: understanding disparities in advance care planning","volume":"77","author":"Ashana","journal-title":"J Gerontol A Biol Sci Med Sci"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Barber-Fleming VA Mann M Mead G Gleeson A . The effectiveness of advance care planning training for care home staff: a systematic review. medRxiv. Preprint posted online on Mar 1, 2023. [doi: 10.1101\/2023.02.28.23286494]","DOI":"10.1101\/2023.02.28.23286494"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-019-4192-0","article-title":"A systematic review of the effects of advance care planning facilitators training programs","volume":"19","author":"Chan","journal-title":"BMC Health Serv Res"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijnurstu.2024.104778","article-title":"Interventions to promote readiness for advance care planning: a systematic review and meta-analysis","volume":"156","author":"Tan","journal-title":"Int J Nurs Stud"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s41999-026-01416-y","article-title":"Evaluation of a novel educational intervention for mental health staff on advance care planning with older people with mental illness","volume":"17","author":"Crous","journal-title":"Eur Geriatr Med"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1038\/s44222-025-00279-5","article-title":"Application of large language models in medicine","volume":"3","author":"Liu","journal-title":"Nat Rev Bioeng"},{"issue":"4","key":"R14","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1038\/s44360-025-00024-7","article-title":"Large language models in global health","volume":"1","author":"Ong","journal-title":"Nat Health"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/j.jpainsymman.2024.11.016","article-title":"Large language models to identify advance care planning in patients with advanced cancer","volume":"69","author":"Agaronnik","journal-title":"J Pain Symptom Manage"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2025.103315","article-title":"Leveraging artificial intelligence in advance care planning: a scoping review","volume":"172","author":"Tan","journal-title":"Artif Intell Med"},{"key":"R18","unstructured":"Russell S Norvig P . Artificial Intelligence: A Modern Approach. Pearson; 2016. ISBN: 9781292153964"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","DOI":"10.1056\/AIp2401106","article-title":"Development, Evaluation, and Assessment of Large Language Models (DEAL) checklist: a technical report","volume":"2","author":"Tripathi","journal-title":"NEJM AI"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1007\/s41666-026-00229-9","article-title":"A scoping review of synthetic data generation by language models in biomedical research and application: data utility and quality perspectives","volume":"10","author":"Rao","journal-title":"J Healthc Inform Res"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1533508","article-title":"Large language models generating synthetic clinical datasets: a feasibility and comparative analysis with real-world perioperative data","volume":"8","author":"Barr","journal-title":"Front Artif Intell"},{"issue":"9","key":"R22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","journal-title":"ACM Comput Surv"},{"key":"R23","unstructured":"Wang Z . Construction of the core knowledge system of advance care planning for community residents. Zhengzhou University. 2021. URL: https:\/\/link.oversea.cnki.net\/doi\/10.27466\/d.cnki.gzzdu.2021.002720 [Accessed 05-06-2026]"},{"key":"R24","unstructured":"Tu T Palepu A Schaekermann M Saab K Freyberg J Tanno R et al. Towards conversational diagnostic AI. arXiv. Preprint posted online on Jan 11, 2024. [doi: 10.48550\/arXiv.2401.05654]"},{"key":"R25","unstructured":"Qwen2.5: a party of foundation models! Qwen Studio. 2024. URL: https:\/\/qwen.ai\/blog?id=qwen2.5 [Accessed 05-06-2026]"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1016\/j.pec.2008.07.016","article-title":"Audit and feedback using the brief Decision Support Analysis Tool (DSAT-10) to evaluate nurse-standardized patient encounters","volume":"73","author":"Stacey","journal-title":"Patient Educ Couns"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Yang S Zhao H Zhu S et al. Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In: Wooldridge M Dy J Natarajan S , editors. AAAI\u201924\/IAAI\u201924\/EAAI\u201924: Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence. AAAI Press; 2024:19368-19376. [doi: 10.1609\/aaai.v38i17.29907]","DOI":"10.1609\/aaai.v38i17.29907"},{"key":"R28","unstructured":"Yang S Zhao H Zhu S Zhou G Xu H Jia Y . Zhongjing. GitHub. 2023. URL: https:\/\/github.com\/SupritYoung\/Zhongjing [Accessed 05-06-2026]"},{"key":"R29","unstructured":"GPT-4o mini: advancing cost-efficient intelligence. OpenAI. 2024. URL: https:\/\/openai.com\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence [Accessed 05-06-2026]"},{"key":"R30","unstructured":"Hu EJ Shen Y Wallis P Allen-Zhu Z Li Y Wang S et al. LoRA: low-rank adaptation of large language models. arXiv. Preprint posted online on Jun 17, 2021. [doi: 10.48550\/arXiv.2106.09685]"},{"key":"R31","unstructured":"Lund BD . A brief review of chatgpt: its value and the underlying GPT technology. arXiv. Preprint posted online on 2023. URL: https:\/\/www.researchgate.net\/publication\/366809571_A_Brief_Review_of_ChatGPT_Its_Value_and_the_Underlying_GPT_Technology [doi: 10.13140\/RG.2.2.28474.06087\/1]"},{"key":"R32","doi-asserted-by":"crossref","unstructured":"Marvin G Hellen N Jjingo D Nakatumba-Nabende J . Prompt engineering in large language models. In: Jacob IJ Piramuthu S Falkowski-Gilski P , editors. Data Intelligence and Cognitive Informatics. Springer; 2024. [doi: 10.1007\/978-981-99-7962-2_30]","DOI":"10.1007\/978-981-99-7962-2_30"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"10","DOI":"10.7275\/pdz9-th90","article-title":"The Delphi technique: making sense of consensus","volume":"12","author":"Hsu","journal-title":"Pract Assess Res Eval"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"issue":"6","key":"R35","doi-asserted-by":"publisher","first-page":"836","DOI":"10.1016\/j.cptl.2015.08.001","article-title":"How to analyze Likert and other rating scale data","volume":"7","author":"Harpe","journal-title":"Curr Pharm Teach Learn"},{"issue":"5","key":"R36","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1007\/s10459-010-9222-y","article-title":"Likert scales, levels of measurement and the \u201claws\u201d of statistics","volume":"15","author":"Norman","journal-title":"Adv Health Sci Educ Theory Pract"},{"issue":"2","key":"R37","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230163","article-title":"ChatGPT and other large language models are double-edged swords","volume":"307","author":"Shen","journal-title":"Radiology"},{"issue":"7972","key":"R38","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-024-10921-0","article-title":"Large language models in medical and healthcare fields: applications, advances, and challenges","volume":"57","author":"Wang","journal-title":"Artif Intell Rev"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2024.109561","article-title":"Fine-tuning language model embeddings to reveal domain knowledge: an explainable artificial intelligence perspective on medical decision making","volume":"139","author":"Krai\u0161nikovi\u0107","journal-title":"Eng Appl Artif Intell"},{"issue":"8","key":"R41","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"4","key":"R42","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooae114","article-title":"Large language models and synthetic health data: progress and prospects","volume":"7","author":"Smolyak","journal-title":"JAMIA Open"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3358206","article-title":"Socially aware synthetic data generation for suicidal ideation detection using large language models","volume":"12","author":"Ghanadian","journal-title":"IEEE Access"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2022.104227","article-title":"Deep learning for rare disease: a scoping review","volume":"135","author":"Lee","journal-title":"J Biomed Inform"},{"issue":"1","key":"R45","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00927-3","article-title":"Harnessing the power of synthetic data in healthcare: innovation, application, and privacy","volume":"6","author":"Giuffr\u00e8","journal-title":"NPJ Digit Med"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.2196\/59505","article-title":"Multimodal large language models in health care: applications, challenges, and future outlook","volume":"26","author":"AlSaad","journal-title":"J Med Internet Res"},{"issue":"1","key":"R47","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-64827-6","article-title":"OpenMedLM: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models","volume":"14","author":"Maharjan","journal-title":"Sci Rep"},{"issue":"14","key":"R48","doi-asserted-by":"publisher","first-page":"6254","DOI":"10.3390\/app14146254","article-title":"Comparing fine-tuning and prompt engineering for multi-class classification in hospitality review analysis","volume":"14","author":"Botunac","journal-title":"Appl Sci"},{"issue":"1","key":"R49","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbad493","article-title":"Opportunities and challenges for ChatGPT and large language models in biomedicine and health","volume":"25","author":"Tian","journal-title":"Brief Bioinformatics"},{"issue":"2","key":"R50","doi-asserted-by":"publisher","first-page":"210","DOI":"10.7326\/M23-2772","article-title":"Large language models in medicine: the potentials and pitfalls: a narrative review","volume":"177","author":"Omiye","journal-title":"Ann Intern Med"},{"issue":"4","key":"R51","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1177\/0825859718763644","article-title":"Advance care planning in Chinese seniors: cultural perspectives","volume":"33","author":"Cheng","journal-title":"J Palliat Care"},{"issue":"1","key":"R52","doi-asserted-by":"publisher","DOI":"10.1186\/s12910-025-01261-x","article-title":"Cultural and ethical barriers to implementing end-of-life advance care planning among oncology nursing professionals: a content analysis of open-ended questions","volume":"26","author":"Shih","journal-title":"BMC Med Ethics"},{"issue":"1","key":"R53","doi-asserted-by":"publisher","DOI":"10.1111\/ggi.70306","article-title":"The effects of advance care planning interventions in the older population: a systematic review and meta\u2010analysis","volume":"26","author":"Lin","journal-title":"Geriatr Gerontol Int"},{"issue":"6","key":"R54","doi-asserted-by":"publisher","first-page":"e333","DOI":"10.1016\/S2589-7500(23)00083-3","article-title":"Ethics of large language models in medicine and medical research","volume":"5","author":"Li","journal-title":"Lancet Digit Health"},{"issue":"1","key":"R55","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01157-x","article-title":"The ethics of ChatGPT in medicine and healthcare: a systematic review on large language models (LLMs)","volume":"7","author":"Haltaufderheide","journal-title":"NPJ Digit Med"},{"issue":"6","key":"R56","doi-asserted-by":"publisher","first-page":"e428","DOI":"10.1016\/S2589-7500(24)00061-X","article-title":"Ethical and regulatory challenges of large language models in medicine","volume":"6","author":"Ong","journal-title":"Lancet Digit Health"},{"key":"R57","unstructured":"HuggingFace. Cassiopeia 2026. URL: https:\/\/huggingface.co\/Cassiopeia2026\/Zhongjing-ACP [Accessed 15-06-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e87288","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:16Z","timestamp":1785536716000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e87288"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,28]]},"references-count":57,"URL":"https:\/\/doi.org\/10.2196\/87288","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,28]]},"article-number":"v28i9e87288"}}