{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T17:45:05Z","timestamp":1793727905454,"version":"4.6.1"},"reference-count":24,"publisher":"Public Library of Science (PLoS)","issue":"4","license":[{"start":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T00:00:00Z","timestamp":1775606400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005230","name":"Chongqing Natural Science Foundation","doi-asserted-by":"crossref","award":["CSTB2024NSCQ-KJFZMSX0065"],"award-info":[{"award-number":["CSTB2024NSCQ-KJFZMSX0065"]}],"id":[{"id":"10.13039\/501100005230","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Jiangsu Province Hospital","award":["TS202401, CZ0121002010039"],"award-info":[{"award-number":["TS202401, CZ0121002010039"]}]},{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["82372019"],"award-info":[{"award-number":["82372019"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The cross-lingual and question-type variations affecting large language models (LLMs) accuracy on the Chinese national medical licensing educations remain insufficiently explored.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In this cross-sectional study (May 13\u201320, 2025), 396 educational questions (198 English\u2013Chinese pairs) were extracted from the Chinese national medical licensing examination. ChatGPT-4o, ChatGPT-o3, Gemini-2.5-pro, Deepseek-V3, Deepseek-R1, and Doubao-1.5-pro were prompted to provide answers. Responses were compared against reference answers, and accuracy was computed for three question types: basic knowledge (Type A), case analysis (Type B), and integrative judgment (Type C).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Across all question types and languages, Doubao-1.5-pro achieved the highest accuracy at 92.0%\u2009\u00b1\u20091.3%, whereas ChatGPT-4o had the lowest accuracy at 82.8%\u2009\u00b1\u20093.7%. There was a significant main effect of question type (\n                      <jats:italic>P<\/jats:italic>\n                      \u2009=\u20090.0038) but no main effect of language (\n                      <jats:italic>P<\/jats:italic>\n                      \u2009=\u20090.56). Post hoc tests confirmed that Type A performance exceeded Types B and C (\n                      <jats:italic>P<\/jats:italic>\n                      \u2009&lt;\u20090.01), while B vs. C did not differ. Among the models, Doubao-1.5-pro, Deepseek-R1, and Deepseek-V3 demonstrated notable cross-lingual stability, with accuracy differences between Chinese and English versions remaining below 5%.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The question type was a key factor affecting LLMs performance on Chinese medical licensing exam questions, whereas language had no significant impact. Doubao-1.5-pro, Deepseek-R1, and Deepseek-V3 demonstrated particularly strong cross-lingual consistency. These findings point to the potential value of specialized LLMs for enhancing medical education in China.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0346518","type":"journal-article","created":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T17:32:22Z","timestamp":1775669542000},"page":"e0346518","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":7,"title":["Performance benchmarking of LLMs on Chinese national medical licensing education: Cross-lingual and question-type effects"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0040-6145","authenticated-orcid":true,"given":"Yuxia","family":"Tang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jian","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shouju","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,4,8]]},"reference":[{"key":"pone.0346518.ref001","doi-asserted-by":"crossref","DOI":"10.2196\/50945","article-title":"The role of large language models in medical education: applications and implications","volume":"9","author":"CW Safranek","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"2","key":"pone.0346518.ref002","doi-asserted-by":"crossref","first-page":"111","DOI":"10.4258\/hir.2025.31.2.111","article-title":"Large language models in medicine","volume":"31","author":"J Choi","year":"2025","journal-title":"Healthc Inform Res"},{"issue":"2","key":"pone.0346518.ref003","doi-asserted-by":"crossref","first-page":"114","DOI":"10.4258\/hir.2025.31.2.114","article-title":"Large language models in medicine: Clinical applications, technical challenges, and ethical considerations","volume":"31","author":"KH Jung","year":"2025","journal-title":"Healthc Inform Res"},{"issue":"7972","key":"pone.0346518.ref004","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"K Singhal","year":"2023","journal-title":"Nature"},{"issue":"2","key":"pone.0346518.ref005","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"TH Kung","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"12","key":"pone.0346518.ref006","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.46721","article-title":"Performance of Large Language Models on a Neurology Board-Style Examination","volume":"6","author":"MC Schubert","year":"2023","journal-title":"JAMA Netw Open"},{"key":"pone.0346518.ref007","doi-asserted-by":"crossref","DOI":"10.2196\/46885","article-title":"The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers","volume":"9","author":"G Eysenbach","year":"2023","journal-title":"JMIR Med Educ"},{"key":"pone.0346518.ref008","article-title":"DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning","author":"GD DeepSeek-AI","year":"2025","journal-title":"arXiv"},{"issue":"8","key":"pone.0346518.ref009","doi-asserted-by":"crossref","first-page":"2550","DOI":"10.1038\/s41591-025-03726-3","article-title":"Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning","volume":"31","author":"M Tordjman","year":"2025","journal-title":"Nat Med"},{"issue":"1","key":"pone.0346518.ref010","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1186\/s12909-024-05125-7","article-title":"Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses","volume":"24","author":"H Zong","year":"2024","journal-title":"BMC Med Educ"},{"key":"pone.0346518.ref011","doi-asserted-by":"crossref","DOI":"10.2196\/66114","article-title":"Large language models in worldwide medical exams: platform development and comprehensive analysis","volume":"26","author":"H Zong","year":"2024","journal-title":"J Med Internet Res"},{"key":"pone.0346518.ref012","doi-asserted-by":"crossref","DOI":"10.2196\/77978","article-title":"Multiple large language models\u2019 performance on the Chinese medical licensing examination: quantitative comparative study","volume":"12","author":"Y Diao","year":"2025","journal-title":"JMIR Hum Factors"},{"issue":"1","key":"pone.0346518.ref013","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1007\/s10916-025-02316-7","article-title":"Towards A Fair Duel: reflections on the evaluation of DeepSeek-R1 and ChatGPT-4o in Chinese medical education","volume":"49","author":"S Li","year":"2025","journal-title":"J Med Syst"},{"key":"pone.0346518.ref014","doi-asserted-by":"crossref","DOI":"10.2196\/52784","article-title":"Influence of Model Evolution and System Roles on ChatGPT\u2019s Performance in Chinese Medical Licensing Exams: Comparative Study","volume":"10","author":"S Ming","year":"2024","journal-title":"JMIR Med Educ"},{"key":"pone.0346518.ref015","doi-asserted-by":"crossref","DOI":"10.2196\/58897","article-title":"Performance of ChatGPT-4 on Taiwanese Traditional Chinese Medicine Licensing Examinations: Cross-Sectional Study","volume":"11","author":"LW Tseng","year":"2025","journal-title":"JMIR Med Educ"},{"issue":"3","key":"pone.0346518.ref016","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1016\/j.identj.2023.12.007","article-title":"Performance of generative artificial intelligence in dental licensing examinations","volume":"74","author":"RCW Chau","year":"2024","journal-title":"Int Dent J"},{"issue":"12","key":"pone.0346518.ref017","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000397","article-title":"How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language","volume":"2","author":"C Fang","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"6","key":"pone.0346518.ref018","doi-asserted-by":"crossref","first-page":"1460","DOI":"10.3390\/clinpract13060130","article-title":"Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine","volume":"13","author":"F Guillen-Grima","year":"2023","journal-title":"Clin Pract"},{"key":"pone.0346518.ref019","doi-asserted-by":"crossref","first-page":"1237432","DOI":"10.3389\/fmed.2023.1237432","article-title":"Artificial intelligence in global health equity: an evaluation and discussion on the application of ChatGPT, in the Chinese National Medical Licensing Examination","volume":"10","author":"W Tong","year":"2023","journal-title":"Front Med (Lausanne)"},{"key":"pone.0346518.ref020","article-title":"Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis","volume":"26","author":"M Liu","year":"2024","journal-title":"J Med Internet Res"},{"issue":"1","key":"pone.0346518.ref021","doi-asserted-by":"crossref","first-page":"20512","DOI":"10.1038\/s41598-023-46995-z","article-title":"Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination","volume":"13","author":"M Roso\u0142","year":"2023","journal-title":"Sci Rep"},{"key":"pone.0346518.ref022","doi-asserted-by":"crossref","first-page":"105173","DOI":"10.1016\/j.ijmedinf.2023.105173","article-title":"Performance and exploration of ChatGPT in medical examination, records and education in Chinese: Pave the way for medical AI","volume":"177","author":"H Wang","year":"2023","journal-title":"Int J Med Inform"},{"issue":"1","key":"pone.0346518.ref023","doi-asserted-by":"crossref","first-page":"1630","DOI":"10.1186\/s12903-025-07035-6","article-title":"Cross-lingual performance of large language models in maxillofacial prosthodontics: a comparative evaluation","volume":"25","author":"I Sozen Yanik","year":"2025","journal-title":"BMC Oral Health"},{"key":"pone.0346518.ref024","article-title":"Performance of ChatGPT-4o and four open-source large language models in generating diagnoses based on china\u2019s rare disease catalog: comparative study","volume":"27","author":"Zhong\u00a0W","year":"2025","journal-title":"J Med Internet Res"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0346518","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T17:32:28Z","timestamp":1775669548000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0346518"}},"subtitle":[],"editor":[{"given":"Mohmed Isaqali","family":"Karobari","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,4,8]]},"references-count":24,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,4,8]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0346518","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,8]]}}}