{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T16:54:16Z","timestamp":1790873656226,"version":"4.1.0"},"reference-count":20,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T00:00:00Z","timestamp":1783468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) perform well on medical examinations, but they are almost always evaluated as test-takers, judged on the accuracy of their answers. Far less is known about their reliability as assessment-automation tools, for instance in extracting examination metadata, or about whether single-run, accuracy-based benchmarks can adequately characterize such tools.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Twelve mainstream LLMs (8 domestic, 4 international) were evaluated on extracting eight metadata fields from a 60-item endocrinology examination, each performing the identical task in three independent runs under realistic web-based conditions. Performance was separated into completion rate, conditional accuracy, and an all-fields-correct task success rate (TSR), with cross-run variance treated as a primary outcome. Conditional accuracy and TSR were compared within models, and domestic versus international models were compared on TSR.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Conditional accuracy was uniformly high, yet TSR was sharply bimodal: five models scored 90% or above and seven below 70%, with none in between. Reliability, not mean accuracy, was decisive. Several models with perfect conditional accuracy collapsed when one or two of their three runs failed entirely, a pattern that single-run evaluation would miss. Failures spanned both model behaviour and platform-level constraints, the latter not removable by prompting. Model origin did not predict performance. On one item, four models independently changed the examiner\u2019s assigned cognitive level to one of their own, a \u201csilent relabeling\u201d with direct implications for item-bank integrity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>For automated assessment metadata extraction, reliability rather than accuracy determines whether an LLM can be deployed, and national origin is not a meaningful predictor. Such tools should be evaluated by repeated runs reporting cross-run variance, and adopted through task-specific piloting within a workflow that reserves human judgement for semantic and normative fields.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2026.1832829","type":"journal-article","created":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T05:58:05Z","timestamp":1783490285000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Beyond accuracy: evaluating the reliability of large language models for medical assessment"],"prefix":"10.3389","volume":"9","author":[{"given":"Hui","family":"Zhang","sequence":"first","affiliation":[{"name":"Office of Academic Affairs, Guangzhou Medical University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lihui","family":"Qu","sequence":"additional","affiliation":[{"name":"Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongbo","family":"Bai","sequence":"additional","affiliation":[{"name":"Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Youbang","family":"Chen","sequence":"additional","affiliation":[{"name":"Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruiying","family":"Ji","sequence":"additional","affiliation":[{"name":"Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zirui","family":"Cheng","sequence":"additional","affiliation":[{"name":"Zhongshan School of Medicine, Sun Yat-Sen University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chun-tao","family":"Yang","sequence":"additional","affiliation":[{"name":"Guangzhou Municipal and Guangdong Provincial Key Laboratory of Protein Modification and Disease, Department of Physiology, School of Basic Medical Sciences, Guangzhou Medical University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,7,8]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"e78432","DOI":"10.2196\/78432","article-title":"Medical feature extraction from clinical examination notes: development and evaluation of a two-phase large language model framework","volume":"13","author":"Abumelha","year":"2025","journal-title":"JMIR Med. Inform."},{"key":"ref2","doi-asserted-by":"publisher","first-page":"2550751","DOI":"10.1080\/10872981.2025.2550751","article-title":"Evaluating large language models as graders of medical short answer questions: a comparative analysis with expert human graders","volume":"30","author":"Bolgova","year":"2025","journal-title":"Med. Educ. Online"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"24086","DOI":"10.48550\/arXiv.2509.24086","article-title":"Do repetitions matter? Strengthening reliability in LLM evaluations","author":"Gonzalez","year":"2025","journal-title":"arXiv"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"e72839","DOI":"10.2196\/72839","article-title":"ChatGPT versus DeepSeek for breast cancer information retrieval: quantitative comparative study","volume":"12","author":"Hajjo","year":"2026","journal-title":"JMIR Cancer."},{"key":"ref5","year":"2024"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1007\/s00404-026-08358-7","article-title":"Performance of next-generation AI chatbots in gynecological knowledge assessment: a comparative pilot study of ChatGPT-5, Gemini-3, DeepSeek-V3.2, and Claude-4.5-opus","volume":"313","author":"Huan","year":"2026","journal-title":"Arch. Gynecol. Obstet."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"e79939","DOI":"10.2196\/79939","article-title":"Application of AI-generated content in medical education: systematic review of the impact on critical thinking abilities of medical students","volume":"12","author":"Li","year":"2026","journal-title":"JMIR Med Educ."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"15868","DOI":"10.48550\/arXiv.2602.15868","article-title":"Understanding LLM failures: a multi-tape turing machine analysis of systematic errors in language model reasoning","author":"Magnus","year":"2026","journal-title":"arXiv"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"100900","DOI":"10.1016\/j.landig.2025.100900","article-title":"How can artificial intelligence transform the training of medical students and physicians?","volume":"7","author":"Ning","year":"2025","journal-title":"Lancet Digit Health."},{"key":"ref10","doi-asserted-by":"publisher","first-page":"83","DOI":"10.61071\/JDP.2555","article-title":"Beyond automation: a conceptual framework for AI in educational assessment","volume":"4","author":"Rare\u0219","year":"2025","journal-title":"J Digit Pedagog"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"e2371","DOI":"10.52225\/narra.v5i1.2371","article-title":"Chinese generative AI models (DeepSeek and Qwen) rival ChatGPT-4 in ophthalmology queries with excellent performance in Arabic and English","volume":"5","author":"Sallam","year":"2025","journal-title":"Narra J"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"1658316","DOI":"10.3389\/frai.2025.1658316","article-title":"Prompt engineering for accurate statistical reasoning with large language models in medical research","volume":"8","author":"Vilakati","year":"2025","journal-title":"Front Artif Intell."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"2054","DOI":"10.1093\/jamia\/ocae079","article-title":"Large language models leverage external knowledge to extend clinical insight beyond language boundaries","volume":"31","author":"Wu","year":"2024","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"e81264","DOI":"10.2196\/81264","article-title":"Comparing AI-assisted problem-solving ability with internet search engine and E-books in medical students with variable prior subject knowledge: cross-sectional study","volume":"12","author":"Xavier","year":"2026","journal-title":"JMIR Med Educ."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"332","DOI":"10.1111\/eje.13073","article-title":"Evaluating the performance of large language models (LLMs) in answering and analysing the chinese dental licensing examination","volume":"29","author":"Xiong","year":"2025","journal-title":"Eur. J. Dent. Educ."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"e69485","DOI":"10.2196\/69485","article-title":"Performance of large language models in the non-English context: qualitative study of models trained on different languages in Chinese medical examinations","volume":"13","author":"Yao","year":"2025","journal-title":"JMIR Med. Inform."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"1313","DOI":"10.12677\/CES.2022.106211","article-title":"Construction of mathematical analysis question bank based on superstar learning under the background of separation of teaching and examination","volume":"10","author":"Zhang","year":"2022","journal-title":"Creat Educ Stud"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"2764","DOI":"10.1038\/s41598-025-32602-w","article-title":"Benchmarking large language models against clinicians across hospital levels in cardiovascular decision-making: a cross-sectional vignette-based study","volume":"16","author":"Zhang","year":"2025","journal-title":"Sci. Rep."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"898","DOI":"10.1038\/s41597-026-07261-9","article-title":"A dataset for evaluating large language models on Chinese national medical licensing examinations","volume":"13","author":"Zong","year":"2026","journal-title":"Sci. Data"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"e66114","DOI":"10.2196\/66114","article-title":"Large language models in worldwide medical exams: platform development and comprehensive analysis","volume":"26","author":"Zong","year":"2024","journal-title":"J. Med. Internet Res."}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1832829\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T05:58:06Z","timestamp":1783490286000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1832829\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,8]]},"references-count":20,"alternative-id":["10.3389\/frai.2026.1832829"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1832829","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,8]]},"article-number":"1832829"}}