{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T09:33:14Z","timestamp":1792143194548,"version":"4.4.0"},"reference-count":42,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n History-taking is crucial in medical training. However, current methods often lack consistent feedback and standardized evaluation and have limited access to standardized patient (SP) resources. Artificial intelligence (AI)\u2013powered simulated patients offer a promising solution; however, challenges such as human-AI consistency, evaluation stability, and transparency remain underexplored in multicase clinical scenarios.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop and validate the AI-Powered Medical History-Taking Training and Evaluation System (AMTES), based on DeepSeek-V2.5 (DeepSeek), to assess its stability, human-AI consistency, and transparency in clinical scenarios with varying symptoms and difficulty levels.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We developed AMTES, a system using multiple strategies to ensure dialog quality and automated assessment. A prospective study with 31 medical students evaluated AMTES\u2019s performance across 3 cases of varying complexity: a simple case (cough), a moderate case (frequent urination), and a complex case (abdominal pain). To validate our design, we conducted systematic baseline comparisons to measure the incremental improvements from each level of our design approach and tested the framework\u2019s generalizability by implementing it with an alternative large language model (LLM) Qwen-Max (Qwen AI; version 20250409), under a zero-modification condition.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 31 students practiced with our AMTES. During the training, students generated 8606 questions across 93 history-taking sessions. AMTES achieved high dialog accuracy: 98.6% (SD 1.5%) for cough, 99.0% (SD 1.1%) for frequent urination, and 97.9% (SD 2.2%) for abdominal pain, with contextual appropriateness exceeding 99%. The system\u2019s automated assessments demonstrated exceptional stability and high human-AI consistency, supported by transparent, evidence-based rationales. Specifically, the coefficients of variation (CV) were low across total scores (0.87%\u20101.12%) and item-level scoring (0.55%\u20100.73%). Total score consistency was robust, with the intraclass correlation coefficients (ICCs) exceeding 0.923 across all scenarios, showing strong agreement. The item-level consistency was remarkably high, consistently above 95%, even for complex cases like abdominal pain (95.75% consistency). In systematic baseline comparisons, the fully-processed system improved ICCs from 0.414\/0.500 to 0.923\/0.972 (moderate and complex cases), with all CVs \u22641.2% across the 3 cases. A zero-modification implementation of our evaluation framework with an alternative LLM (Qwen-Max) achieved near-identical performance, with the item-level consistency rates over 94.5% and ICCs exceeding 0.89. Overall, 87% of students found AMTES helpful, and 83% expressed a desire to use it again in the future.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Our data showed that AMTES demonstrates significant educational value through its LLM-based virtual SPs, which successfully provided authentic clinical dialogs with high response accuracy and delivered consistent, transparent educational feedback. Combined with strong user approval, these findings highlight AMTES\u2019s potential as a valuable, adaptable, and generalizable tool for medical history-taking training across various educational contexts.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/73419","type":"journal-article","created":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T22:19:59Z","timestamp":1756505999000},"page":"e73419-e73419","source":"Crossref","is-referenced-by-count":21,"title":["Development and Validation of a Large Language Model\u2013Based System for Medical History-Taking Training: Prospective Multicase Study on Evaluation Stability, Human-AI Consistency, and Transparency"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-3818-2967","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-0126-9311","authenticated-orcid":false,"given":"Chujun","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7725-0668","authenticated-orcid":false,"given":"Liping","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7925-7125","authenticated-orcid":false,"given":"Xiule","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1721-7635","authenticated-orcid":false,"given":"Xiaoqin","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6584-7157","authenticated-orcid":false,"given":"Yiying","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9410-2961","authenticated-orcid":false,"given":"Haizhu","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7324-9872","authenticated-orcid":false,"given":"Weishan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,8,29]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-015-0443-x","article-title":"Teaching history taking to medical students: a systematic review","volume":"15","author":"Keifenheim","journal-title":"BMC Med Educ"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1016\/j.ejim.2006.10.002","article-title":"Core competencies of the European internist: A discussion paper","volume":"18","author":"Palsson","journal-title":"Eur J Intern Med"},{"key":"R3","doi-asserted-by":"crossref","unstructured":"Alharbi L Almoallim H . History-taking skills in rheumatology. In: Almoallim H Cheikh M , editors. Skills in Rheumatology. Springer; 2021:3-16.","DOI":"10.1007\/978-981-15-8323-0_1"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.3389\/fped.2022.850912","article-title":"History taking as a diagnostic tool in children with chronic cough","volume":"10","author":"Kantar","journal-title":"Front Pediatr"},{"issue":"7","key":"R5","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1007\/s00063-020-00731-x","article-title":"Medical history taking and clinical examination in emergency and intensive care medicine","volume":"115","author":"Steinkellner","journal-title":"Med Klin Intensivmed Notfmed"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.15052","article-title":"A clinical medicine level test at Jinan University School of Medicine reveals the importance of training medical students in clinical history-taking","volume":"11","author":"Meng","journal-title":"PeerJ"},{"key":"R7","unstructured":"Aranda JH Monks SM . Roles and responsibilities of the standardized patient director in medical simulation. In: StatPearls. StatPearls Publishing LLC; 2024."},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.4414\/smw.2020.20382","article-title":"An overview of and approach to selecting appropriate patient representations in teaching and summative assessment in medical education","volume":"150","author":"Bauer","journal-title":"Swiss Med Wkly"},{"issue":"12","key":"R9","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1111\/medu.13387","article-title":"Understanding the impact of simulated patients on health care learners\u2019 communication skills: a systematic review","volume":"51","author":"Kaplonyi","journal-title":"Med Educ"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.nedt.2022.105312","article-title":"History-taking level and its influencing factors among nursing undergraduates based on the virtual standardized patient testing results: Cross sectional study","volume":"111","author":"Du","journal-title":"Nurse Educ Today"},{"issue":"3","key":"R11","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1111\/j.1365-2923.2008.03268.x","article-title":"Feedback by simulated patients in undergraduate medical education: a systematic review of the literature","volume":"43","author":"Bokken","journal-title":"Med Educ"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/53961","article-title":"A generative pretrained transformer (GPT)-powered chatbot as a simulated patient to practice history taking: prospective, mixed methods study","volume":"10","author":"Holderried","journal-title":"JMIR Med Educ"},{"issue":"5","key":"R13","doi-asserted-by":"publisher","first-page":"926","DOI":"10.1002\/ase.2270","article-title":"The rise of ChatGPT: Exploring its potential in medical education","volume":"17","author":"Lee","journal-title":"Anat Sci Educ"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/47274","article-title":"The intersection of ChatGPT, clinical medicine, and medical education","volume":"9","author":"Wong","journal-title":"JMIR Med Educ"},{"issue":"3","key":"R15","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1097\/ACM.0000000000004963","article-title":"Competencies for the use of artificial intelligence-based tools by health care professionals","volume":"98","author":"Russell","journal-title":"Acad Med"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.2196\/51411","article-title":"Medical education and artificial intelligence: web of science-based bibliometric analysis (2013-2022)","volume":"10","author":"Wang","journal-title":"JMIR Med Educ"},{"issue":"5","key":"R17","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.38755","article-title":"Proof of concept: using ChatGPT to teach emergency physicians how to break bad news","volume":"15","author":"Webb","journal-title":"Cureus"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/68486","article-title":"Virtual patients using large language models: scalable, contextualized simulation of clinician-patient dialogue with feedback","volume":"27","author":"Cook","journal-title":"J Med Internet Res"},{"issue":"Suppl 3","key":"R19","doi-asserted-by":"publisher","first-page":"1988","DOI":"10.1002\/jdd.13591","article-title":"Using artificial intelligence chatbots to improve patient history taking in dental education (Pilot study)","volume":"88 Suppl 3","author":"Or","journal-title":"J Dent Educ"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/59435","article-title":"Application of large language models in medical training evaluation-using ChatGPT as a standardized patient: multimetric assessment","volume":"27","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"9","key":"R21","doi-asserted-by":"publisher","first-page":"1346","DOI":"10.1097\/ACM.0000000000001578","article-title":"Feedback for learners in medical education: what is known? A scoping review","volume":"92","author":"Bing-You","journal-title":"Acad Med"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/59213","article-title":"A language model-powered simulated patient with automated feedback for history taking: prospective study","volume":"10","author":"Holderried","journal-title":"JMIR Med Educ"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2024.2445037","article-title":"Artificial intelligence in health professions education assessment: AMEE guide no. 178","author":"Masters","journal-title":"Med Teach"},{"issue":"3","key":"R24","doi-asserted-by":"publisher","first-page":"209","DOI":"10.3857\/roj.2023.00633","article-title":"Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer","volume":"41","author":"Choi","journal-title":"Radiat Oncol J"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"187","DOI":"10.4274\/dir.2024.242892","article-title":"A retrospective evaluation of the potential of ChatGPT in the accurate diagnosis of acute stroke","volume":"31","author":"Kuzan","journal-title":"Diagn Interv Radiol"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/50174","article-title":"ChatGPT in medical education: a precursor for automation bias?","volume":"10","author":"Nguyen","journal-title":"JMIR Med Educ"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.2196\/57674","article-title":"Data set and benchmark (MedGPTEval) to evaluate responses from large language models in medicine: evaluation development and validation","volume":"12","author":"Xu","journal-title":"JMIR Med Inform"},{"issue":"10","key":"R28","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of chatbot responses to physician questions","volume":"6","author":"Goodman","journal-title":"JAMA Netw Open"},{"issue":"4","key":"R29","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1097\/SIH.0000000000000373","article-title":"Virtual standardized patient simulation: case development and pilot application to high-value care","volume":"14","author":"Bond","journal-title":"Simul Healthc"},{"issue":"2","key":"R30","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1097\/SIH.0000000000000195","article-title":"Developing a conversational virtual standardized patient to enable students to practice history-taking skills","volume":"12","author":"Maicher","journal-title":"Simul Healthc"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-05982-2","article-title":"Analysis of virtual standardized patients for assessing clinical fundamental skills of medical students: a prospective study","volume":"24","author":"Zhang","journal-title":"BMC Med Educ"},{"issue":"2","key":"R32","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1111\/j.1365-2923.2007.02973.x","article-title":"What is feedback in clinical education?","volume":"42","author":"van de Ridder","journal-title":"Med Educ"},{"issue":"6","key":"R33","doi-asserted-by":"publisher","first-page":"528","DOI":"10.1111\/medu.14075","article-title":"Between trust and control: Teachers\u2019 assessment conceptualisations within programmatic assessment","volume":"54","author":"Schut","journal-title":"Med Educ"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics14010109","article-title":"A systematic review and meta-analysis of artificial intelligence tools in medicine and healthcare: applications, considerations, limitations, motivation and challenges","volume":"14","author":"Younis","journal-title":"Diagnostics (Basel)"},{"issue":"7","key":"R35","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1016\/j.cptl.2019.03.009","article-title":"Use of virtual patient software to assess student confidence and ability in communication skills and virtual patient impression: A mixed-methods approach","volume":"11","author":"Borja-Hart","journal-title":"Curr Pharm Teach Learn"},{"issue":"7","key":"R36","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1016\/j.cptl.2020.02.008","article-title":"Use of a virtual patient simulation program to enhance the physical assessment and medical history taking skills of doctor of pharmacy students","volume":"12","author":"Fidler","journal-title":"Curr Pharm Teach Learn"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/55927","article-title":"Benchmarking state-of-the-art large language models for migraine patient education: performance comparison of responses to common queries","volume":"26","author":"Li","journal-title":"J Med Internet Res"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.2196\/70703","article-title":"AI in home care-evaluation of large language models for future training of informal caregivers: observational comparative case study","volume":"27","author":"P\u00e9rez-Esteve","journal-title":"J Med Internet Res"},{"issue":"12","key":"R39","doi-asserted-by":"publisher","first-page":"1440","DOI":"10.1080\/0142159X.2023.2247144","article-title":"Teaching clinical empathy skills in medical education: Can ChatGPT assist the educator?","volume":"45","author":"Ennab","journal-title":"Med Teach"},{"issue":"9258","key":"R40","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1016\/s0140-6736(00)04169-6","article-title":"Influence of context effects on health outcomes: a systematic review","volume":"357","author":"Di Blasi","journal-title":"Lancet"},{"issue":"6","key":"R41","doi-asserted-by":"publisher","first-page":"869","DOI":"10.1089\/107555303771952226","article-title":"Empathy, enablement, and outcome: an exploratory study on acupuncture patients\u2019 perceptions","volume":"9","author":"MacPherson","journal-title":"J Altern Complement Med"},{"issue":"4","key":"R42","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1089\/acm.2005.11.591","article-title":"A pilot prospective study on the consultation and relational empathy, patient enablement, and health changes over 12 months in patients going to the Glasgow Homoeopathic Hospital","volume":"11","author":"Bikker","journal-title":"J Altern Complement Med"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e73419","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:04:08Z","timestamp":1782842648000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e73419"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,29]]},"references-count":42,"URL":"https:\/\/doi.org\/10.2196\/73419","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,29]]},"article-number":"v11i2e73419"}}