{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T16:47:03Z","timestamp":1791046023126,"version":"4.1.0"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Mock examinations are widely used in health professional education to assess learning and prepare candidates for national licensure. However, instructor-written multiple-choice items can vary in difficulty, coverage, and clarity. Recently, large language models (LLMs) have achieved high accuracy in medical examinations, highlighting their potential for assisting item-bank development; however, their educational quality remains insufficiently characterized.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to (1) identify the most accurate LLM for the Japanese National Examination for Radiological Technologists and (2) use the top model to generate blueprint-aligned multiple-choice questions and evaluate their educational quality.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Four LLMs\u2014OpenAI o3, o4-mini, o4-mini-high (OpenAI), and Gemini 2.5 Flash (Google)\u2014were evaluated on all 200 items of the 77th Japanese National Examination for Radiological Technologists in 2025. Accuracy was analyzed for overall items and for 173 nonimage items. The best-performing model (o3) then generated 192 original items across 14 subjects by matching the official blueprint (image-based items were excluded). Subject-matter experts (\u22655 y as coordinators and routine mock examination authors) independently rated each generated item on five criteria using a 5-point scale (1=unacceptable, 5=adoptable): item difficulty, factual accuracy, accuracy of content coverage, appropriateness of wording, and instructional usefulness. Cochran Q with Bonferroni-adjusted McNemar tests compared model accuracies, and one-sided Wilcoxon signed-rank tests assessed whether the median ratings exceeded 4.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n OpenAI o3 achieved the highest accuracy overall (90.0%; 95% CI 85.1%\u201093.4%) and on nonimage items (92.5%; 95% CI 87.6%\u201095.6%), significantly outperforming o4-mini on the full set (\n P<\/jats:italic>\n =.02). Across models, accuracy differences on the non-image subset were not significant (Cochran Q,\n P<\/jats:italic>\n =.10). Using o3, the 192 generated items received high expert ratings for item difficulty (mean, 4.29; 95% CI 4.11\u20104.46), factual accuracy (4.18; 95% CI 3.98\u20104.38), and content coverage (4.73; 95% CI 4.60\u20104.86). Ratings were comparatively lower for appropriateness of wording (3.92; 95% CI 3.73\u20104.11) and instructional usefulness (3.60; 95% CI 3.41\u20103.80). For these two criteria, the tests did not support a median rating >4 (one-sided Wilcoxon,\n P<\/jats:italic>\n =.45 and\n P<\/jats:italic>\n \u2265.99, respectively). Representative low-rated examples (ratings 1\u20102) and the rationale for those scores\u2014such as ambiguous phrasing or generic explanations without linkage to stem cues\u2014are provided in the supplementary materials.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n OpenAI o3 can generate radiological licensure items that align with national standards in terms of difficulty, factual correctness, and blueprint coverage. However, wording clarity and the pedagogical specificity of explanations were weaker and did not meet an adoptable threshold without further editorial refinement. These findings support a practical workflow in which LLMs draft syllabus-aligned items at scale, while faculty perform targeted edits to ensure clarity and formative feedback. Future studies should evaluate image-inclusive generation, use Application Programming Interface (API)-pinned model snapshots to increase reproducibility, and develop guidance to improve explanation quality for learner remediation.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/81807","type":"journal-article","created":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T15:15:21Z","timestamp":1761664521000},"page":"e81807-e81807","source":"Crossref","is-referenced-by-count":4,"title":["Large Language Models for the National Radiological Technologist Licensure Examination in Japan: Cross-Sectional Comparative Benchmarking and Evaluation of Model-Generated Items Study"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5613-2801","authenticated-orcid":false,"given":"Toshimune","family":"Ito","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8114-5097","authenticated-orcid":false,"given":"Toru","family":"Ishibashi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5269-4081","authenticated-orcid":false,"given":"Tatsuya","family":"Hayashi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6199-9784","authenticated-orcid":false,"given":"Shinya","family":"Kojima","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4457-536X","authenticated-orcid":false,"given":"Kazumi","family":"Sogabe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,11,13]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"157","DOI":"10.4103\/sjmms.sjmms_173_21","article-title":"Do mock medical licensure exams improve performance of graduates? Experience from a Saudi Medical College","volume":"10","author":"Al-Sheikh","journal-title":"Saudi J Med Sci"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1177\/23821205231206221","article-title":"Impact of an online question bank on Resident In-Training exam performance","volume":"10","author":"Scott","journal-title":"J Med Educ Curric Dev"},{"issue":"4","key":"R3","doi-asserted-by":"publisher","first-page":"58","DOI":"10.22159\/ijcpr.2020v12i4.39079","article-title":"Pharmacy students\u2019 opinions of using mock questions to prepare for summative examinations","volume":"12","author":"Siab","journal-title":"Int J Curr Pharm Sci"},{"issue":"80","key":"R4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.37376\/jsh.vi80.6626","article-title":"An evaluation of a final year multiple choice questions examination at Faculty of medicine-university of Benghazi","volume":"12","author":"Alawgali","journal-title":"Open Access Maced J Med Sci"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1186\/s12909-019-1544-8","article-title":"Barriers and facilitators to writing quality items for medical school assessments \u2013 a scoping review","volume":"19","author":"Karthikeyan","journal-title":"BMC Med Educ"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000433","article-title":"Performance of generative pretrained transformer on the National Medical Licensing Examination in Japan","volume":"3","author":"Tanaka","journal-title":"PLOS Digit Health"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.3352\/jeehp.2025.22.16","article-title":"Performance of large language models on Thailand\u2019s National Medical Licensing Examination: a cross-sectional study","volume":"22","author":"Saowaprut","journal-title":"J Educ Eval Health Prof"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.2196\/63731","article-title":"Qwen-2.5 outperforms other large language models in the Chinese National Nursing Licensing Examination: retrospective cross-sectional comparative study","volume":"13","author":"Zhu","journal-title":"JMIR Med Inform"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-79245-x","article-title":"Leveraging large language models to construct feedback from medical multiple-choice questions","volume":"14","author":"Tomova","journal-title":"Sci Rep"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"919","DOI":"10.1007\/s40670-024-02251-1","article-title":"Pilot study on using large language models for educational resource development in Japanese Radiological Technologist Exams","volume":"35","author":"Kondo","journal-title":"MedSciEduc"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajoint.2024.100070","article-title":"The performance of artificial intelligence-based large language models on ophthalmology-related questions in Swedish proficiency test for medicine: ChatGPT-4 omni vs Gemini 1.5 Pro","volume":"1","author":"Sabaner","journal-title":"AJO International"},{"issue":"9","key":"R13","doi-asserted-by":"publisher","first-page":"3872","DOI":"10.1016\/j.acra.2024.06.046","article-title":"Large language models as tools to generate radiology board-style multiple-choice questions","volume":"31","author":"Mistry","journal-title":"Acad Radiol"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Brin D Sorin V Konen E Nadkarni G Glicksberg BS Klang E . How large language models perform on the united states medical licensing examination: a systematic review. medRxiv. Preprint posted online on 2023. [doi: 10.1101\/2023.09.03.23294842]","DOI":"10.1101\/2023.09.03.23294842"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/66114","article-title":"Large language models in worldwide medical exams: platform development and comprehensive analysis","volume":"26","author":"Zong","journal-title":"J Med Internet Res"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-46995-z","article-title":"Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination","volume":"13","author":"Roso\u0142","journal-title":"Sci Rep"},{"issue":"6","key":"R17","doi-asserted-by":"publisher","first-page":"9020","DOI":"10.5489\/cuaj.9020","article-title":"Use of AI (GPT-4)-generated multiple-choice questions for the examination of surgical subspecialty residents","volume":"19","author":"Kim","journal-title":"CUAJ"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.metrad.2024.100103","article-title":"Potential of multimodal large language models for data mining of medical images and free-text reports","volume":"2","author":"Zhang","journal-title":"Meta-Radiology"},{"issue":"7","key":"R19","doi-asserted-by":"publisher","DOI":"10.3174\/ajnr.A8589","article-title":"A review of the opportunities and challenges with large language models in radiology: the road ahead","volume":"46","author":"Soni","journal-title":"AJNR Am J Neuroradiol"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2025.01.001","article-title":"MiniMedGPT: efficient large vision\u2013language model for Medical Visual Question Answering","volume":"189","author":"Alsabbagh","journal-title":"Pattern Recognit Lett"},{"key":"R21","unstructured":"Chen J Shen X Li X Elhoseiny M . MiniGPT-4: enhancing vision-language understanding with advanced large language models. arXiv. Preprint posted online on Oct 2, 2023. [doi: 10.48550\/arXiv.2304.10592]"},{"key":"R22","unstructured":"Zhang P Zang Y et al. InternLM-xcomposer2-4KHD: a pioneering large vision-language model handling resolutions from 336 pixels to 4K HD. Adv Neural Inf Process Syst. Preprint posted online on Apr 9, 2024. [doi: 10.48550\/arXiv.2404.06512]"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Wang Z Huang Y Wu Y et al. Fusion side tuning: a parameter and memory efficient fine-tuning method for high-resolution medical image classification. Presented at: 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Dec 3-6, 2024; IEEE. Lisbon, Portugal. [doi: 10.1109\/BIBM62325.2024.10821946] [Medline: 40989005]","DOI":"10.1109\/BIBM62325.2024.10821946"},{"key":"R24","unstructured":"Gemini Team GoogleGeorgiev P Lei VI et al. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv. Preprint posted online on Dec 16, 2024. [doi: 10.48550\/arXiv.2403.05530]"},{"issue":"10","key":"R25","doi-asserted-by":"publisher","first-page":"1881","DOI":"10.1111\/ijd.17729","article-title":"Diagnostic performance of GPT-4o and Gemini Flash 2.0 in acne and rosacea","volume":"64","author":"Boostani","journal-title":"Int J Dermatol"},{"issue":"02","key":"R26","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1055\/s-0044-1792040","article-title":"Evaluating ChatGPT-4\u2019s performance in Identifying Radiological Anatomy in FRCR Part 1 Examination Questions","volume":"35","author":"Sarangi","journal-title":"Indian J Radiol Imaging"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1055\/s-0043-1777746","article-title":"Assessing the capability of ChatGPT, Google Bard, and Microsoft Bing in solving Radiology case vignettes","volume":"34","author":"Sarangi","journal-title":"Indian J Radiol Imaging"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Morse K Kumar A et al. Assessing the potential of USMLE-like exam questions generated by GPT-4. medRxiv. Preprint posted online on Apr 28, 2023. [doi: 10.1101\/2023.04.25.23288588]","DOI":"10.1101\/2023.04.25.23288588"},{"issue":"5","key":"R29","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1067\/j.cpradiol.2025.02.002","article-title":"Differentiating between GPT-generated and human-written feedback for radiology residents","volume":"54","author":"Zhou","journal-title":"Curr Probl Diagn Radiol"},{"issue":"6","key":"R30","doi-asserted-by":"publisher","first-page":"1538","DOI":"10.46328\/ijemst.4440","article-title":"Assessing AI-generated (GPT-4) versus human created MCQs In Mathematics education: a comparative inquiry into vector topics","volume":"12","author":"Kuusemets","journal-title":"IJEMST"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Ma W Yang C K\u00e4stner C . (Why) is my prompt getting worse? Rethinking regression testing for evolving LLM APIs. Presented at: Proceedings of the IEEE\/ACM 3rd International Conference on AI Engineering-Software Engineering for AI; Apr 14, 2024:166-171; Lisbon Portugal. [doi: 10.1145\/3644815.3644950]","DOI":"10.1145\/3644815.3644950"},{"key":"R32","unstructured":"Schroeder K Wood-Doughty Z . Can you trust LLM judgments? Reliability of LLM-as-a-judge. arXiv. Preprint posted online on Feb 18, 2025. [doi: 10.48550\/arXiv.2412.12509] [Medline: 38076521]"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Renze M . The effect of sampling temperature on problem solving in large language models. In: Al-Onaizan Y Bansal M Chen YN , editors. Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics; 2024:7346-7356. URL: https:\/\/aclanthology.org\/2024.findings-emnlp [Accessed 01-08-2025] [doi: 10.18653\/v1\/2024.findings-emnlp.432]","DOI":"10.18653\/v1\/2024.findings-emnlp.432"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e81807","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:15:42Z","timestamp":1782843342000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2025\/1\/e81807"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,13]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/81807","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,13]]},"article-number":"v11i2e81807"}}