{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T11:59:32Z","timestamp":1785153572464,"version":"3.55.0"},"reference-count":29,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T00:00:00Z","timestamp":1785110400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) have shown promise in medical education, but most are trained primarily on Western medical data and lack sufficient exposure to Traditional Chinese Medicine (TCM). Despite growing interest in LLMs for TCM education, systematic evaluations of their performance remain limited. However, it remains unclear at which cognitive levels\u2014from basic recall to clinical reasoning\u2014these models can reliably support learners, and whether their multimodal capabilities meet the demands of authentic TCM pedagogical tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to evaluate LLMs\u2019 performance on authentic Chinese medicine knowledge tasks, including both text-based and image-based assessments, to provide evidence for their potential application in TCM education.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      Seven LLMs were evaluated across four task types: textual objective questions, textual subjective syndrome differentiation tasks, image-based objective herb identification, and image-based subjective tongue diagnosis. For objective tasks, performance was measured by accuracy. For subjective tasks, performance was evaluated using natural language processing metrics and blinded expert ratings. Statistical analyses included Cochran\u2019s\n                      <jats:italic>Q<\/jats:italic>\n                      test, Friedman rank-sum test, and Bonferroni-corrected\n                      <jats:italic>post-hoc<\/jats:italic>\n                      comparisons. Inter-rater reliability was verified using the intraclass correlation coefficient.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Significant tiered performance differences emerged across 7 models. For textual objective questions, Doubao-1.5-Pro achieved the highest accuracy (91.23%), with performance declining markedly from lower-order memorization to higher-order clinical reasoning tasks. For textual subjective tasks, SparkDesk-v3.5 (94.11%), Doubao-1.5-Pro (93.03%), and Gemini-3-Preview (92.99%) formed the top tier. A notable subjective-objective performance inversion was identified: SparkDesk-v3.5 ranked lowest in objective accuracy but highest in subjective scoring. Our exploratory image-based results suggest that no model exceeded 60% accuracy in herbal identification (\n                      <jats:italic>N<\/jats:italic>\n                      \u202f=\u202f55), while tongue diagnosis performance was comparatively higher but should also be interpreted cautiously (\n                      <jats:italic>N<\/jats:italic>\n                      \u202f=\u202f223). Qualitative analysis using a four-category error typology revealed visual misidentification, cross-modal association failures, terminological-level errors, and reasoning-level inconsistencies.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Top-tier large language models demonstrate potential as virtual teaching assistants for foundational Traditional Chinese Medicine knowledge navigation. However, their deployment in complex syndrome differentiation and multimodal teaching poses substantial risks due to model errors and insufficient clinical reasoning. Image-based findings should be considered preliminary and require validation in larger, more balanced multimodal datasets. Clearly defined applicability boundaries, rigorous manual review procedures, and continuous enhancement of domain-specific multimodal datasets are essential before integration into educational practice.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fmed.2026.1893231","type":"journal-article","created":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T11:10:29Z","timestamp":1785150629000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Are AI tutors ready for the TCM classroom? A multimodal evaluation of large language models across cognitive levels in Chinese medicine education"],"prefix":"10.3389","volume":"13","author":[{"given":"Jie","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Public Administration, School of Management, Beijing University of Chinese Medicine","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruojia","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Public Administration, School of Management, Beijing University of Chinese Medicine","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fengying","family":"Guo","sequence":"additional","affiliation":[{"name":"Department of Public Administration, School of Management, Beijing University of Chinese Medicine","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifan","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Public Administration, School of Management, Beijing University of Chinese Medicine","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,7,27]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"S12","DOI":"10.1038\/d41586-023-02918-6","article-title":"How AI is transforming medical education","volume":"621","author":"Savage","year":"2023","journal-title":"Nature"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-026-01631-5","article-title":"A triaxial benchmark for assessing responses from large language models in traditional Chinese medicine","author":"Huang","year":"2026","journal-title":"Commun Med"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"466","DOI":"10.1038\/s41746-025-01845-2","article-title":"Evaluating the role of large language models in traditional Chinese medicine diagnosis and treatment recommendations","volume":"8","author":"Liu","year":"2025","journal-title":"npj Digit Med"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"1667104","DOI":"10.3389\/fmed.2025.1667104","article-title":"Supporting postgraduate exam preparation with large language models: implications for traditional Chinese medicine education","volume":"12","author":"Wang","year":"2025","journal-title":"Front Med"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"19","DOI":"10.13193\/j.issn.1673-7717.2025.10.004","article-title":"Comparative study of generative large language models for professional competencies in the field of traditional Chinese medicine","volume":"43","author":"Zhang","year":"2025","journal-title":"Chin Arch Tradit Chin Med"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"e14658","DOI":"10.2196\/14658","article-title":"A virtual counseling application using artificial intelligence for communication skills training in nursing education: development study","volume":"21","author":"Shorey","year":"2019","journal-title":"J Med Internet Res"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"ref11","first-page":"110","volume-title":"Findings of the Association for Computational Linguistics: ACL","author":"Liu","year":"2023"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1111\/opo.12131","article-title":"When to use the Bonferroni correction","volume":"34","author":"Armstrong","year":"2014","journal-title":"Ophthalmic Physiol Opt"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"e35179","DOI":"10.7759\/cureus.35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"key":"ref15","first-page":"610","author":"Bender","year":"2021"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1801829","DOI":"10.3389\/fpubh.2026.1801829","article-title":"Challenges of using generative AI for patient education in chronic heart failure: an evaluation of content quality, readability, and actionability in cross-platform LLM-generated texts","volume":"14","author":"Wang","year":"2026","journal-title":"Front Public Health"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"1828810","DOI":"10.3389\/fcvm.2026.1828810","article-title":"Cross-platform evaluation of LLM-generated educational texts on cardiac myxoma: quality, readability, and actionability using network analysis and latent profile analysis","volume":"13","author":"Deng","year":"2026","journal-title":"Front Cardiovasc Med"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"887","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns","volume":"11","author":"Sallam","year":"2023","journal-title":"Healthcare (Basel)"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","year":"2023","journal-title":"Nature"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"1773","DOI":"10.1038\/s41591-022-01981-2","article-title":"Multimodal biomedical AI","volume":"28","author":"Acosta","year":"2022","journal-title":"Nat Med"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"2290123","DOI":"10.1080\/10872981.2023.2290123","article-title":"The illusion of knowledge: generative AI and the loss of tacit clinical skills","volume":"29","author":"Michael","year":"2024","journal-title":"Med Educ Online"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"e230045","DOI":"10.1148\/ryai.230045","article-title":"Vision-language models in medical imaging: a narrative review","volume":"5","author":"Kanjee","year":"2023","journal-title":"Radiol Artif Intell"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"976","DOI":"10.1080\/0142159X.2019.1595557","article-title":"Artificial intelligence in medical education","volume":"41","author":"Masters","year":"2019","journal-title":"Med Teach"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1016\/0959-4752(94)90003-5","article-title":"Cognitive load theory, learning difficulty, and instructional design","volume":"4","author":"Sweller","year":"1994","journal-title":"Learn Instr"},{"key":"ref26","doi-asserted-by":"crossref","DOI":"10.4324\/9780203994627","volume-title":"The Logic of Scientific Discovery","author":"Popper","year":"2005"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"e46885","DOI":"10.2196\/46885","article-title":"The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers","volume":"9","author":"Eysenbach","year":"2023","journal-title":"JMIR Med Educ"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat Med"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"104598","DOI":"10.1016\/j.jbi.2024.104598","article-title":"Multimodal large language models for medical applications: a comprehensive review","volume":"150","author":"Li","year":"2024","journal-title":"J Biomed Inform"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1893231\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T11:10:31Z","timestamp":1785150631000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1893231\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,27]]},"references-count":29,"alternative-id":["10.3389\/fmed.2026.1893231"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1893231","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,27]]},"article-number":"1893231"}}