{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:53:19Z","timestamp":1791568399310,"version":"4.3.4"},"reference-count":46,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T00:00:00Z","timestamp":1791504000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["32270690"],"award-info":[{"award-number":["32270690"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012542","name":"Sichuan Province Science and Technology Support Program","doi-asserted-by":"publisher","award":["2024YFHZ0205"],"award-info":[{"award-number":["2024YFHZ0205"]}],"id":[{"id":"10.13039\/100012542","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["32570773"],"award-info":[{"award-number":["32570773"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Large language models (LLMs) such as GPT-4 and DeepSeek have shown potential in medical education and assessment. In China, the Chinese National Medical Licensing Examination (CNMLE) has become a critical benchmark for evaluating LLMs\u2019 professional knowledge and reasoning ability in non-English contexts. This scoping review aims to synthesize research evaluating LLM performance on the CNMLE and related Chinese medical examinations, identifying performance trends, methodological gaps, and future directions. A scoping review was conducted in accordance with PRISMA-ScR guidelines. PubMed and Web of Science were searched on June 12, 2025, using terms related to LLMs and Chinese medical examinations. Studies were included if they evaluated any LLM on national-level Chinese medical exams and reported performance metrics. Two reviewers screened and extracted data. Study quality was assessed using a 12-item checklist covering dataset characteristics, LLM setup, and evaluation methods. Fisher\u2019s exact test was used to assess differences in the passing rates of different LLMs. 14 studies were included, covering 51 evaluation records across 8 types of Chinese medical examinations, including CNMLE and specialty exams such as critical care, radiation oncology, and ultrasound medicine. Exam years ranged from 2017 to 2024, with a shift toward using recent exams. A total of 9 LLMs were evaluated, including GPT-3.5, GPT-4, GPT-4o, ERNIE, DeepSeek-R1, Qwen-72B, Baichuan2-7B, Baichuan2-13B and DISC-MedLLM. GPT-4o and DeepSeek-R1 achieved the highest scores of 552 (92.00%) and 523 (87.20%) in the 2024 CNMLE, respectively, both surpassing the passing threshold. In an exploratory comparison, GPT-4 achieved a higher pass rate than GPT-3.5 with a statistically significant difference, although inconsistencies in datasets, model transparency, and evaluation criteria persist across the included studies. LLMs show improving performance on Chinese medical exams, especially newer models. Nevertheless, future research should prioritize standardized benchmarks, multimodal capabilities, and transparent evaluation to ensure meaningful clinical relevance and educational value.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001773","type":"journal-article","created":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:22:22Z","timestamp":1791566542000},"page":"e0001773","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Performance of large language models in the Chinese National Medical Licensing Examinations and Beyond: Scoping review"],"prefix":"10.1371","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9142-5017","authenticated-orcid":true,"given":"Hui","family":"Zong","sequence":"first","affiliation":[{"name":"Joint Laboratory of Artificial Intelligence for Critical Care Medicine","place":["Chengdu, China"],"department":["Department of Critical Care Medicine and Institutes for Systems Genetics"]},{"name":"Centre for Medical Big Data and Artificial Intelligence, The First Affiliated Hospital (Southwest Hospital) of Army Medical University (Third Military Medical University)","place":["Chongqing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiao","family":"Wang","sequence":"additional","affiliation":[{"name":"Joint Laboratory of Artificial Intelligence for Critical Care Medicine","place":["Chengdu, China"],"department":["Department of Critical Care Medicine and Institutes for Systems Genetics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaxue","family":"Cha","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of Signaling and Disease Research, School of Life Sciences and Technology, Tongji University","place":["Shanghai, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rongrong","family":"Wu","sequence":"additional","affiliation":[{"name":"Joint Laboratory of Artificial Intelligence for Critical Care Medicine","place":["Chengdu, China"],"department":["Department of Critical Care Medicine and Institutes for Systems Genetics"]},{"name":"The First Affiliated Hospital of Soochow University","place":["Suzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shanshan","family":"Hu","sequence":"additional","affiliation":[{"name":"Joint Laboratory of Artificial Intelligence for Critical Care Medicine","place":["Chengdu, China"],"department":["Department of Critical Care Medicine and Institutes for Systems Genetics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan","family":"Zhao","sequence":"additional","affiliation":[{"name":"The First Affiliated Hospital (Southwest Hospital) of Army Medical University (Third Military Medical University)","place":["Chongqing, China"],"department":["Department of Prevention Healthcare"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2899-1531","authenticated-orcid":true,"given":"Bairong","family":"Shen","sequence":"additional","affiliation":[{"name":"Joint Laboratory of Artificial Intelligence for Critical Care Medicine","place":["Chengdu, China"],"department":["Department of Critical Care Medicine and Institutes for Systems Genetics"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,10,9]]},"reference":[{"key":"pdig.0001773.ref001","article-title":"Benchmark evaluation of DeepSeek large language models in clinical decision-making","author":"S Sandmann","year":"2025","journal-title":"Nat Med"},{"key":"pdig.0001773.ref002","article-title":"Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning","author":"M Tordjman","year":"2025","journal-title":"Nat Med"},{"issue":"2","key":"pdig.0001773.ref003","doi-asserted-by":"crossref","DOI":"10.1056\/AIcs2300145","article-title":"Using ChatGPT to facilitate truly informed medical consent","volume":"1","author":"FN Mirza","year":"2024","journal-title":"NEJM AI"},{"key":"pdig.0001773.ref004","unstructured":"Yang A, Li A, Yang B, Zhang B, Hui B, Zheng B. Qwen3 technical report. 2025. https:\/\/arxiv.org\/abs\/250509388"},{"key":"pdig.0001773.ref005","doi-asserted-by":"crossref","first-page":"104716","DOI":"10.1016\/j.jbi.2024.104716","article-title":"Advancing Chinese biomedical text mining with community challenges","volume":"157","author":"H Zong","year":"2024","journal-title":"J Biomed Inform"},{"issue":"11","key":"pdig.0001773.ref006","doi-asserted-by":"crossref","DOI":"10.1056\/AIe2400881","article-title":"We need more randomized clinical trials of AI","volume":"1","author":"D Ouyang","year":"2024","journal-title":"NEJM AI"},{"issue":"10","key":"pdig.0001773.ref007","doi-asserted-by":"crossref","first-page":"2878","DOI":"10.1038\/s41591-024-03148-7","article-title":"Outpatient reception via collaboration between nurses and a large language model: A randomized controlled trial","volume":"30","author":"P Wan","year":"2024","journal-title":"Nat Med"},{"issue":"4","key":"pdig.0001773.ref008","doi-asserted-by":"crossref","DOI":"10.1056\/AIe2400146","article-title":"Trials of AI interventions must be preregistered","volume":"1","author":"JM Drazen","year":"2024","journal-title":"NEJM AI"},{"key":"pdig.0001773.ref009","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1201\/9781003508595-2","article-title":"Artificial intelligence in improving disease diagnosis: A case study of cardiovascular disease prediction","volume-title":"Artificial Intelligence in Medicine and Healthcare","author":"A Pati","year":"2025"},{"key":"pdig.0001773.ref010","article-title":"Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis","volume":"26","author":"M Liu","year":"2024","journal-title":"J Med Internet Res"},{"issue":"2","key":"pdig.0001773.ref011","article-title":"It\u2019s time to bench the medical exam benchmark","volume":"2","author":"ID Raji","year":"2025","journal-title":"NEJM AI"},{"key":"pdig.0001773.ref012","doi-asserted-by":"crossref","DOI":"10.2196\/66114","article-title":"Large language models in worldwide medical exams: Platform development and comprehensive analysis","volume":"26","author":"H Zong","year":"2024","journal-title":"J Med Internet Res"},{"issue":"1","key":"pdig.0001773.ref013","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1186\/s12909-022-03385-9","article-title":"Experiences, challenges, and prospects of National Medical Licensing Examination in China","volume":"22","author":"X Wang","year":"2022","journal-title":"BMC Med Educ"},{"issue":"1","key":"pdig.0001773.ref014","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1186\/s12909-024-05125-7","article-title":"Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses","volume":"24","author":"H Zong","year":"2024","journal-title":"BMC Med Educ"},{"issue":"1","key":"pdig.0001773.ref015","doi-asserted-by":"crossref","first-page":"14119","DOI":"10.1038\/s41598-025-98949-2","article-title":"Evaluating the performance of GPT-3.5, GPT-4, and GPT-4o in the Chinese National Medical Licensing Examination","volume":"15","author":"D Luo","year":"2025","journal-title":"Sci Rep"},{"issue":"1","key":"pdig.0001773.ref016","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1007\/s10916-025-02213-z","article-title":"Performance of DeepSeek-R1 and ChatGPT-4o on the Chinese National Medical Licensing Examination: A Comparative Study","volume":"49","author":"J Wu","year":"2025","journal-title":"J Med Syst"},{"key":"pdig.0001773.ref017","article-title":"A comparative analysis of GPT-4o and ERNIE Bot in a Chinese radiation oncology exam","author":"W Wang","year":"2025","journal-title":"J Cancer Educ"},{"key":"pdig.0001773.ref018","doi-asserted-by":"crossref","first-page":"155010","DOI":"10.1016\/j.jcrc.2024.155010","article-title":"Can ChatGPT-4 perform as a competent physician based on the Chinese critical care examination?","volume":"86","author":"X Wang","year":"2025","journal-title":"J Crit Care"},{"key":"pdig.0001773.ref019","doi-asserted-by":"crossref","first-page":"1472006","DOI":"10.3389\/fmed.2024.1472006","article-title":"The performance of AI in medical examinations: An exploration of ChatGPT in ultrasound medical education","volume":"11","author":"D-R Hong","year":"2024","journal-title":"Front Med (Lausanne)"},{"issue":"7","key":"pdig.0001773.ref020","doi-asserted-by":"crossref","first-page":"467","DOI":"10.7326\/M18-0850","article-title":"PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation","volume":"169","author":"AC Tricco","year":"2018","journal-title":"Ann Intern Med"},{"key":"pdig.0001773.ref021","doi-asserted-by":"crossref","first-page":"104988","DOI":"10.1016\/j.ebiom.2024.104988","article-title":"PubMed and beyond: Biomedical literature search in the age of artificial intelligence","volume":"100","author":"Q Jin","year":"2024","journal-title":"EBioMedicine"},{"issue":"1","key":"pdig.0001773.ref022","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/s11192-015-1765-5","article-title":"The journal coverage of Web of Science and Scopus: A comparative analysis","volume":"106","author":"P Mongeon","year":"2015","journal-title":"Scientometrics"},{"key":"pdig.0001773.ref023","doi-asserted-by":"crossref","first-page":"1442975","DOI":"10.3389\/frai.2024.1442975","article-title":"Evaluation of large language models under different training background in Chinese medical examination: A comparative study","volume":"7","author":"S Zhang","year":"2024","journal-title":"Front Artif Intell"},{"key":"pdig.0001773.ref024","doi-asserted-by":"crossref","DOI":"10.2196\/52784","article-title":"Influence of model evolution and system roles on ChatGPT\u2019s performance in Chinese Medical Licensing Exams: Comparative Study","volume":"10","author":"S Ming","year":"2024","journal-title":"JMIR Med Educ"},{"issue":"9","key":"pdig.0001773.ref025","doi-asserted-by":"crossref","first-page":"2054","DOI":"10.1093\/jamia\/ocae079","article-title":"Large language models leverage external knowledge to extend clinical insight beyond language boundaries","volume":"31","author":"J Wu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"12","key":"pdig.0001773.ref026","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000397","article-title":"How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language","volume":"2","author":"C Fang","year":"2023","journal-title":"PLOS Digit Health"},{"key":"pdig.0001773.ref027","doi-asserted-by":"crossref","first-page":"1237432","DOI":"10.3389\/fmed.2023.1237432","article-title":"Artificial intelligence in global health equity: an evaluation and discussion on the application of ChatGPT, in the Chinese National Medical Licensing Examination","volume":"10","author":"W Tong","year":"2023","journal-title":"Front Med (Lausanne)"},{"issue":"12","key":"pdig.0001773.ref028","doi-asserted-by":"crossref","first-page":"6112","DOI":"10.1016\/j.asjsur.2023.09.089","article-title":"Can ChatGPT pass China\u2019s national medical licensing examination?","volume":"46","author":"L Shang","year":"2023","journal-title":"Asian J Surg"},{"issue":"1","key":"pdig.0001773.ref029","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1007\/s10916-023-01961-0","article-title":"ChatGPT performs on the Chinese national medical licensing examination","volume":"47","author":"X Wang","year":"2023","journal-title":"J Med Syst"},{"key":"pdig.0001773.ref030","doi-asserted-by":"crossref","first-page":"105173","DOI":"10.1016\/j.ijmedinf.2023.105173","article-title":"Performance and exploration of ChatGPT in medical examination, records and education in Chinese: Pave the way for medical AI","volume":"177","author":"H Wang","year":"2023","journal-title":"Int J Med Inform"},{"issue":"1","key":"pdig.0001773.ref031","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1186\/s12909-024-05239-y","article-title":"Large language models for generating medical examinations: Systematic review","volume":"24","author":"Y Artsi","year":"2024","journal-title":"BMC Med Educ"},{"issue":"3","key":"pdig.0001773.ref032","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"K Singhal","year":"2025","journal-title":"Nat Med"},{"issue":"2","key":"pdig.0001773.ref033","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"TH Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"pdig.0001773.ref034","article-title":"A survey of large language models in medicine: Progress, application, and challenge","author":"H Zhou","year":"2023","journal-title":"arXiv preprint arXiv:231105112"},{"issue":"6","key":"pdig.0001773.ref035","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1038\/s44222-025-00279-5","article-title":"Application of large language models in medicine","volume":"3","author":"F Liu","year":"2025","journal-title":"Nat Rev Bioeng"},{"issue":"1","key":"pdig.0001773.ref036","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1038\/s41746-023-00952-2","article-title":"A medical multimodal large language model for future pandemics","volume":"6","author":"F Liu","year":"2023","journal-title":"NPJ Digit Med"},{"key":"pdig.0001773.ref037","doi-asserted-by":"crossref","first-page":"52430","DOI":"10.52202\/075280-2283","article-title":"Benchmarking large language models on CMExam: A comprehensive Chinese Medical Exam Dataset","volume":"36","author":"J Liu","year":"2023","journal-title":"Adv Neural Inform Process Syst"},{"key":"pdig.0001773.ref038","article-title":"A dataset for evaluating large language models on Chinese National Medical Licensing Examinations","author":"H Zong","year":"2026","journal-title":"Scientific Data"},{"issue":"1","key":"pdig.0001773.ref039","doi-asserted-by":"crossref","first-page":"5424","DOI":"10.1038\/s41467-025-61531-5","article-title":"Uncovering memorization effect in the presence of spurious correlations","volume":"16","author":"C You","year":"2025","journal-title":"Nat Commun"},{"issue":"4","key":"pdig.0001773.ref040","doi-asserted-by":"crossref","first-page":"1116","DOI":"10.26599\/BDMA.2024.9020044","article-title":"MedBench: A comprehensive, standardized, and reliable benchmarking system for evaluating Chinese Medical Large Language Models","volume":"7","author":"M Liu","year":"2024","journal-title":"Big Data Min Anal"},{"issue":"12","key":"pdig.0001773.ref041","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1097\/RLI.0000000000001005","article-title":"A multiclass radiomics method-based WHO Severity Scale for improving COVID-19 patient assessment and disease characterization from CT scans","volume":"58","author":"JAG Henao","year":"2023","journal-title":"Invest Radiol"},{"issue":"1","key":"pdig.0001773.ref042","doi-asserted-by":"crossref","first-page":"101118","DOI":"10.1016\/j.patter.2024.101118","article-title":"A survey of multilingual large language models","volume":"6","author":"L Qin","year":"2025","journal-title":"Patterns (N Y)"},{"issue":"6","key":"pdig.0001773.ref043","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000877","article-title":"Retrieval augmented generation for large language models in healthcare: A systematic review","volume":"4","author":"LM Amugongo","year":"2025","journal-title":"PLOS Digit Health"},{"key":"pdig.0001773.ref044","article-title":"Auto-encoding knowledge graph for unsupervised medical report generation","volume-title":"Proceedings of the 35th International Conference on Neural Information Processing Systems","author":"F Liu","year":"2021"},{"key":"pdig.0001773.ref045","article-title":"Retrieve, reason, and refine: generating accurate and faithful patient instructions","volume-title":"Proceedings of the 36th International Conference on Neural Information Processing Systems","author":"F Liu","year":"2022"},{"issue":"1","key":"pdig.0001773.ref046","doi-asserted-by":"crossref","first-page":"736","DOI":"10.1186\/s12909-024-05738-y","article-title":"Exploring the potential of artificial intelligence to enhance the writing of english academic papers by non-native english-speaking medical students - the educational application of ChatGPT","volume":"24","author":"J Li","year":"2024","journal-title":"BMC Med Educ"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001773","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:22:31Z","timestamp":1791566551000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001773"}},"subtitle":[],"editor":[{"given":"John","family":"Batani","sequence":"first","affiliation":[{"name":"Botho University","place":["LESOTHO"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,10,9]]},"references-count":46,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10,9]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001773","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,9]]}}}