{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T17:15:13Z","timestamp":1788455713033,"version":"build-2803163510"},"reference-count":28,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language model (LLM)\u2013based AI teaching agents are increasingly used in medical education, yet their pedagogical quality is typically judged by platform-generated scores whose scoring criteria are undisclosed and may not reflect the teaching quality of the agent.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop and validate a multidimensional rubric for evaluating AI teaching agents and to examine the correspondence between platform scores and rubric-based teaching quality.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Eight AI teaching agents covering an endocrinology curriculum were deployed across 4 role-play paradigms (patient, student, expert, and family). Twenty-two fourth-year medical students generated 167 dialogues, which were scored both by the platform and by an independently applied 8-dimension rubric (100 points, covering knowledge accuracy, pedagogical guidance, knowledge coverage, role-play quality, difficulty calibration, medical safety, student engagement, and feedback quality). Each dialogue was scored 4 times by a primary evaluator (Claude Opus 4.8; mean within-model SD 0.36), with 2 additional LLMs as robustness checks; 40 dialogues spanning all agents were rescored by a medical-education expert for validation.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Platform and rubric rankings diverged for most agents: the agent ranked third by the platform ranked last on rubric-based quality, and the platform\u2019s fourth-ranked agent ranked first. Agents differed most on knowledge-related dimensions (knowledge coverage coefficient of variation=27.3%) and least on role-play quality (coefficient of variation=5.7%), while difficulty calibration was a shared weakness. In a case-level observation, one agent revised specifically to strengthen empathy attained high role-play quality yet the lowest knowledge coverage of all agents. AI scores agreed with expert ratings at the total-score level (intraclass correlation coefficient=0.51) and on cognitive-process dimensions, but agreement was low for the more subjective dimensions. Student gender showed no detectable effect, though this analysis was underpowered.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n In this exploratory study, platform-generated scores reflected a construct different from agent teaching quality and should be used as a complement rather than as the sole quality indicator. The 8-dimension rubric provides a transparent, standardized alternative that reveals differences missed by platform scores, including a lack of association between empathy and knowledge coverage that warrants attention in future agent design.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/96819","type":"journal-article","created":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T16:21:36Z","timestamp":1788452496000},"page":"e96819-e96819","source":"Crossref","is-referenced-by-count":0,"title":["What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9924-9695","authenticated-orcid":false,"given":"Hui","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4676-1430","authenticated-orcid":false,"given":"Lihui","family":"Qu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2712-1079","authenticated-orcid":false,"given":"Jianmin","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0469-4396","authenticated-orcid":false,"given":"Yi","family":"Xiong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5439-6901","authenticated-orcid":false,"given":"Hongbo","family":"Bai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2665-5266","authenticated-orcid":false,"given":"Ruiying","family":"Ji","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2923-6947","authenticated-orcid":false,"given":"Guohui","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7545-8057","authenticated-orcid":false,"given":"Wanling","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2894-3062","authenticated-orcid":false,"given":"Zirui","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8901-7014","authenticated-orcid":false,"given":"Youbang","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5640-0930","authenticated-orcid":false,"given":"Chun-tao","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,3]]},"reference":[{"issue":"8","key":"R1","doi-asserted-by":"publisher","first-page":"744","DOI":"10.1016\/j.eprac.2022.04.008","article-title":"The current status of preclinical endocrine education in U.S. medical schools","volume":"28","author":"Long","journal-title":"Endocr Pract"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","first-page":"200","DOI":"10.3760\/cma.j.cn112338-20250512-00309","article-title":"Advantages and challenges of applying artificial intelligence in medical education","volume":"47","author":"Hu","journal-title":"Zhonghua Liu Xing Bing Xue Za Zhi"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-06530-8","article-title":"Enhancing medical education with chatbots: a randomized controlled trial on standardized patients for colorectal cancer","volume":"24","author":"Jiang","journal-title":"BMC Med Educ"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-06877-6","article-title":"AI-powered standardised patients: evaluating ChatGPT-4o\u2019s impact on clinical case management in intern physicians","volume":"25","author":"\u00d6nc\u00fc","journal-title":"BMC Med Educ"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2025.1601116","article-title":"Is the use of standardized patients more effective than role-playing in medical education? A meta-analysis","volume":"12","author":"Xiao","journal-title":"Front Med (Lausanne)"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1097\/JCMA.0000000000000431","article-title":"Role-play of real patients improves the clinical performance of medical students","volume":"84","author":"Huang","journal-title":"J Chin Med Assoc"},{"issue":"7","key":"R7","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.42027","article-title":"Standardized patients in medical education: a review of the literature","volume":"15","author":"Flanagan","journal-title":"Cureus"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01283-x","article-title":"Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education","volume":"6","author":"Yu","journal-title":"Commun Med (Lond)"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-06399-7","article-title":"Large language models improve clinical decision making of medical students through patient simulation and structured feedback: a randomized controlled trial","volume":"24","author":"Br\u00fcgge","journal-title":"BMC Med Educ"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07770-y","article-title":"Effectiveness of AI-assisted medical education for Chinese undergraduate medical students: a meta-analysis","volume":"25","author":"Peng","journal-title":"BMC Med Educ"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s40670-024-02206-6","article-title":"ChatGPT and other large language models in medical education - scoping literature review","volume":"35","author":"Aster","journal-title":"Med Sci Educ"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare13060603","article-title":"A review of large language models in medical education, clinical decision support, and healthcare administration","volume":"13","author":"Vrdoljak","journal-title":"Healthcare (Basel)"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/81271","article-title":"Large language model-based patient simulation to foster communication skills in health care professionals: user-centered development and usability study","volume":"11","author":"Elhilali","journal-title":"JMIR Med Educ"},{"issue":"3","key":"R14","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1093\/jamia\/ocad222","article-title":"Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review","volume":"31","author":"Ding","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.xinn.2025.101253","article-title":"A survey on LLM-as-a-judge","volume":"7","author":"Gu","journal-title":"Innovation (Camb)"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Gorenshtein A Omar M Glicksberg BS Nadkarni GN Klang E . AI agents in clinical medicine: a systematic review. medRxiv. Aug 26, 2025:2025.08.22.25334232. [doi: 10.1101\/2025.08.22.25334232] [Medline: 40909853]","DOI":"10.1101\/2025.08.22.25334232"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2026.1707196","article-title":"Impact of educational agents on student\u2019s learning outcomes: a meta-analysis","volume":"17","author":"Xu","journal-title":"Front Psychol"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","DOI":"10.1111\/tct.70011","article-title":"Pediatric endocrinology education among trainees: a scoping review","volume":"22","author":"Alshammri","journal-title":"Clin Teach"},{"issue":"5","key":"R19","doi-asserted-by":"publisher","first-page":"312","DOI":"10.1007\/s40037-020-00641-w","article-title":"Diabetes SPECIAL (Students Providing Education on Chronic Illness and Lifestyle): a novel preclinical medical student elective","volume":"10","author":"Myers","journal-title":"Perspect Med Educ"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1152\/advan.00101.2019","article-title":"Challenging endocrinology students with a critical-thinking workbook","volume":"44","author":"Daemicke","journal-title":"Adv Physiol Educ"},{"issue":"4","key":"R21","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.82013","article-title":"Scaffolding simulation activities for medical students learning cardiopulmonary assessment: a retrospective study","volume":"17","author":"Hodge","journal-title":"Cureus"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2026.1832829","article-title":"Beyond accuracy: evaluating the reliability of large language models for medical assessment","volume":"9","author":"Zhang","journal-title":"Front Artif Intell"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2025.115804","article-title":"Jagged competencies: measuring the reliability of generative AI in academic research","volume":"203","author":"Thomas","journal-title":"J Bus Res"},{"issue":"18","key":"R24","doi-asserted-by":"publisher","first-page":"2734","DOI":"10.1002\/sim.7679","article-title":"Performance of intraclass correlation coefficient (ICC) as a reliability index under various distributions in scale reliability studies","volume":"37","author":"Mehta","journal-title":"Stat Med"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02005-2","article-title":"Evaluating clinical AI summaries with large language models as judges","volume":"8","author":"Croxford","journal-title":"NPJ Digit Med"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1683322","article-title":"AI for evidence-based treatment recommendation in oncology: a blinded evaluation of large language models and agentic workflows","volume":"8","author":"Zhai","journal-title":"Front Artif Intell"},{"issue":"4","key":"R27","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1152\/advan.00137.2024","article-title":"Accuracy and reliability of large language models in assessing learning outcomes achievement across cognitive domains","volume":"48","author":"Teckwani","journal-title":"Adv Physiol Educ"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1097\/JPA.0000000000000742","article-title":"Artificial intelligence in medical education: promoting active learning with a customized chatbot tool","volume":"37","author":"Bogenschutz","journal-title":"J Physician Assist Educ"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T16:21:38Z","timestamp":1788452498000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e96819"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,3]]},"references-count":28,"URL":"https:\/\/doi.org\/10.2196\/96819","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,3]]},"article-number":"v12i8e96819"}}