{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T12:48:48Z","timestamp":1786970928020,"version":"build-2736575974"},"reference-count":34,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T00:00:00Z","timestamp":1769040000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"\"Zhou's\" nursing research project of The First Affiliated Hospital of Soochow University","award":["HLYJ-Z-202502"],"award-info":[{"award-number":["HLYJ-Z-202502"]}]},{"name":"University-Industry Collaborative Education Program","award":["230905329045253"],"award-info":[{"award-number":["230905329045253"]}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) are increasingly applied in medical education, yet their reliability in specialized, high-stakes assessments such as the Chinese Health Professional and Technical Examination remains unclear. DeepSeek-R1, a recently released reasoning-enhanced LLM, has shown promising performance, but empirical evidence within nursing examination contexts is limited.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare the performance of DeepSeek-R1 and the GPT-4o API on the Chinese Health Professional and Technical Examination (Intermediate Nursing), focusing on accuracy, response consistency, and consistent accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Four hundred official practice examination multiple-choice questions were categorized into four competency units and two question types (A\/B). Both models were evaluated using overall accuracy, consistency (agreement across repeated responses), and consistent accuracy (proportion of responses that were both consistent and correct). Stratified analyses were performed across units, question types, and disciplines. Chi-square tests were used for statistical comparison, and Holm\u2013Bonferroni correction was applied for multiple comparisons.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>DeepSeek-R1 demonstrated significantly higher overall accuracy than the GPT-4o API (88.5% vs. 67.9%, P\u2009&lt;\u20090.001). GPT-4o API showed higher response consistency (96.5% vs. 88.5%) but lower consistent accuracy (66.7% vs. 84.0%). After multiple-comparison correction, significant differences in consistent accuracy remained in basic knowledge, professional knowledge, professional practice ability and Type A questions, as well as in surgical and gynecological nursing disciplines, while other domains showed no statistically significant differences.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec005\">\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>DeepSeek-R1 outperformed the GPT-4o API across multiple dimensions of nursing competency assessment, particularly in overall accuracy and consistent accuracy. GPT-4o API exhibited high response stability but a tendency toward systematic errors, underscoring the need for careful interpretation of model outputs. Further research is needed to evaluate LLM performance using open-ended clinical reasoning tasks and real-world assessment data to support safe and effective educational integration.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0338328","type":"journal-article","created":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T18:28:29Z","timestamp":1769106509000},"page":"e0338328","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":2,"title":["Performance of DeepSeek and ChatGPT on the Chinese Health Professional and Technical Examination: A comparative study"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7576-8741","authenticated-orcid":true,"given":"Xu","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xu","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huiting","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiang","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pin","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hailing","family":"Ju","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,1,22]]},"reference":[{"issue":"3","key":"pone.0338328.ref001","doi-asserted-by":"crossref","DOI":"10.1097\/NNE.0000000000001553","article-title":"Innovative teaching strategies in DNP education using ChatGPT","volume":"49","author":"V Quattrini","year":"2024","journal-title":"Nurse Educ"},{"issue":"6","key":"pone.0338328.ref002","doi-asserted-by":"crossref","DOI":"10.1097\/NNE.0000000000001657","article-title":"OpenAI\u2019s ChatGPT clinical simulation\u00a0: an innovative teaching strategy for clinical nursing education","volume":"49","author":"E Musallam","year":"2024","journal-title":"Nurse Educ"},{"issue":"5","key":"pone.0338328.ref003","doi-asserted-by":"crossref","DOI":"10.1097\/NNE.0000000000001650","article-title":"Harnessing the power of artificial intelligence to guide nurse educator students","volume":"49","author":"LK Woodley","year":"2024","journal-title":"Nurse Educ"},{"issue":"3","key":"pone.0338328.ref004","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1097\/NNE.0000000000001634","article-title":"Work with ChatGPT, not against: 3 teaching strategies that harness the power of artificial intelligence","volume":"49","author":"RC Simms","year":"2024","journal-title":"Nurse Educ"},{"key":"pone.0338328.ref005","doi-asserted-by":"crossref","first-page":"101487","DOI":"10.1016\/j.ecns.2023.101487","article-title":"Enhancing healthcare education: leveraging ChatGPT for innovative simulation scenarios","volume":"87","author":"J Vaughn","year":"2024","journal-title":"Clin Simul Nurs"},{"issue":"8","key":"pone.0338328.ref006","article-title":"Assessing the performance of GPT-3.5 and GPT-4 on the 2023 Japanese nursing examination","volume":"15","author":"Y Kaneda","year":"2023","journal-title":"Cureus"},{"key":"pone.0338328.ref007","article-title":"Performance of the large language model ChatGPT on the national nurse examinations in Japan: evaluation study","volume":"6","author":"K Taira","year":"2023","journal-title":"JMIR Nurs"},{"issue":"21","key":"pone.0338328.ref008","doi-asserted-by":"crossref","first-page":"2855","DOI":"10.3390\/healthcare11212855","article-title":"Performance of ChatGPT on registered nurse license exam in Taiwan: a descriptive study","volume":"11","author":"H Huang","year":"2023","journal-title":"Healthcare (Basel)"},{"key":"pone.0338328.ref009","doi-asserted-by":"crossref","first-page":"104717","DOI":"10.1016\/j.ijnurstu.2024.104717","article-title":"Assessing question characteristic influences on ChatGPT\u2019s performance and response-explanation consistency: insights from Taiwan\u2019s nursing licensing exam","volume":"153","author":"M-C Su","year":"2024","journal-title":"Int J Nurs Stud"},{"issue":"1","key":"pone.0338328.ref010","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1186\/s12909-024-05125-7","article-title":"Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses","volume":"24","author":"H Zong","year":"2024","journal-title":"BMC Med Educ"},{"issue":"4","key":"pone.0338328.ref011","doi-asserted-by":"crossref","first-page":"446","DOI":"10.1080\/0142159X.2024.2314198","article-title":"A scoping review of artificial intelligence in medical education: BEME Guide No. 84","volume":"46","author":"M Gordon","year":"2024","journal-title":"Med Teach"},{"key":"pone.0338328.ref012","doi-asserted-by":"crossref","DOI":"10.2196\/52784","article-title":"Influence of model evolution and system roles on ChatGPT\u2019s performance in Chinese medical licensing exams: comparative study","volume":"10","author":"S Ming","year":"2024","journal-title":"JMIR Med Educ"},{"issue":"1","key":"pone.0338328.ref013","doi-asserted-by":"crossref","DOI":"10.1186\/s12903-025-05554-w","article-title":"ChatGPT-4 Omni\u2019s superiority in answering multiple-choice oral radiology questions","volume":"25","author":"M Tassoker","year":"2025","journal-title":"BMC Oral Health"},{"issue":"8050","key":"pone.0338328.ref014","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1038\/d41586-025-00259-0","article-title":"How China created AI model DeepSeek and shocked the world","volume":"638","author":"G Conroy","year":"2025","journal-title":"Nature"},{"key":"pone.0338328.ref015","doi-asserted-by":"crossref","first-page":"110539","DOI":"10.1016\/j.resuscitation.2025.110539","article-title":"Artificial intelligence in cardiopulmonary resuscitation: driving awareness and debunking myths","volume":"208","author":"F Semeraro","year":"2025","journal-title":"Resuscitation"},{"key":"pone.0338328.ref016","volume-title":"Medical Reasoning in LLMs: An In-Depth Analysis of DeepSeek R12025","author":"B Moell"},{"issue":"8","key":"pone.0338328.ref017","doi-asserted-by":"crossref","first-page":"2550","DOI":"10.1038\/s41591-025-03726-3","article-title":"Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning","volume":"31","author":"M Tordjman","year":"2025","journal-title":"Nat Med"},{"key":"pone.0338328.ref018","article-title":"Reasoning beyond accuracy: expert evaluation of large language models in diagnostic pathology","author":"A Waqas","year":"2025","journal-title":"medRxiv"},{"issue":"20","key":"pone.0338328.ref019","first-page":"4","article-title":"Empirical exploration and prospect of ChatGPT-like large language models in nursing education","volume":"42","author":"L Mu","year":"2024","journal-title":"Health Vocat Educ"},{"issue":"19","key":"pone.0338328.ref020","first-page":"124","article-title":"Scoping review of the application of large language models in nursing","volume":"39","author":"YZ Ma","year":"2024","journal-title":"J Nurs Sci"},{"issue":"1","key":"pone.0338328.ref021","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1186\/s12912-025-03010-2","article-title":"Can large language models facilitate the effective implementation of nursing processes in clinical settings?","volume":"24","author":"Y Cao","year":"2025","journal-title":"BMC Nurs"},{"issue":"5","key":"pone.0338328.ref022","doi-asserted-by":"crossref","first-page":"109713","DOI":"10.1016\/j.isci.2024.109713","article-title":"The application of large language models in medicine: a scoping review","volume":"27","author":"X Meng","year":"2024","journal-title":"iScience"},{"key":"pone.0338328.ref023","article-title":"Performance of ChatGPT on nursing licensure examinations in the United States and China: cross-sectional study","volume":"10","author":"Z Wu","year":"2024","journal-title":"JMIR Med Educ"},{"key":"pone.0338328.ref024","doi-asserted-by":"crossref","DOI":"10.2196\/50336","article-title":"Authors\u2019 reply to: variability in large language models\u2019 responses to medical licensing and certification examinations","volume":"9","author":"A Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"1","key":"pone.0338328.ref025","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1186\/s13054-025-05302-0","article-title":"Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study","volume":"29","author":"JD Workum","year":"2025","journal-title":"Crit Care"},{"issue":"9","key":"pone.0338328.ref026","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"P Hager","year":"2024","journal-title":"Nat Med"},{"issue":"3","key":"pone.0338328.ref027","doi-asserted-by":"crossref","first-page":"104068","DOI":"10.1016\/j.ipm.2025.104068","article-title":"Amplifying commonsense knowledge via bi-directional relation integrated graph-based contrastive pre-training from large language models","volume":"62","author":"L Yu","year":"2025","journal-title":"Inform Process Manage"},{"issue":"2","key":"pone.0338328.ref028","doi-asserted-by":"crossref","first-page":"430","DOI":"10.26599\/BDMA.2024.9020068","article-title":"Federated transfer learning for on-device LLMs efficient fine tuning optimization","volume":"8","author":"C Li","year":"2025","journal-title":"Big Data Min Anal"},{"issue":"1","key":"pone.0338328.ref029","article-title":"Large language models in Traditional Chinese Medicine: a scoping review","volume":"18","author":"Y Ren","year":"2025","journal-title":"J Evid Based Med"},{"issue":"2","key":"pone.0338328.ref030","doi-asserted-by":"crossref","first-page":"55","DOI":"10.3390\/nursrep15020055","article-title":"Exploring applications of artificial intelligence in critical care nursing: a systematic review","volume":"15","author":"E Porcellato","year":"2025","journal-title":"Nurs Rep"},{"issue":"1","key":"pone.0338328.ref031","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1186\/s13040-024-00408-7","article-title":"Comparing new tools of artificial intelligence to the authentic intelligence of our global health students","volume":"17","author":"SR Thandla","year":"2024","journal-title":"BioData Min"},{"issue":"1","key":"pone.0338328.ref032","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1186\/s12911-024-02709-7","article-title":"Analyzing evaluation methods for large language models in the medical field: a scoping review","volume":"24","author":"J Lee","year":"2024","journal-title":"BMC Med Inform Decis Mak"},{"key":"pone.0338328.ref033","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TKDE.2022.3150080","article-title":"Knowledge graph quality management: a comprehensive survey","author":"B Xue","year":"2022","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"1","key":"pone.0338328.ref034","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1186\/s13321-014-0047-1","article-title":"Reliable estimation of prediction errors for QSAR models under model uncertainty using double cross-validation","volume":"6","author":"D Baumann","year":"2014","journal-title":"J Cheminform"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0338328","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T18:28:34Z","timestamp":1769106514000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0338328"}},"subtitle":[],"editor":[{"given":"Zheng","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,1,22]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1,22]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0338328","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,22]]}}}