{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T12:24:51Z","timestamp":1789129491319,"version":"build-2803163510"},"reference-count":41,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"},{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Medical Education and Curricular Development"],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The rapid development of large language models (LLMs) has raised questions about the continued educational value of conventional assessment formats in health professions education. This study evaluated how examination type, search access, image-based question characteristics, and item format influenced LLM accuracy in Japanese national licensing examinations for physicians, dentists, and pharmacists.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Questions from the 2025 academic-year Japanese national licensing examinations administered in early 2026 were analyzed, including 400 medical, 360 dental, and 345 pharmacist questions. Multiple LLMs were assessed under search-on and search-off conditions. Questions were classified by examination type, image presence, image type, and item format. Accuracy was determined using official answer keys, and item-level comparisons were performed using McNemar and chi-square tests. Holm-adjusted sensitivity analyses were conducted to account for multiple comparisons.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Accuracy was highest in the medical examination, intermediate in the pharmacist examination, and lowest in the dental examination. Search access did not consistently improve performance and significantly reduced accuracy in selected model\u2013examination combinations. Image-based questions substantially decreased accuracy, particularly in the dental examination, with reductions of approximately 17 to 28 percentage points; these dental image effects remained significant after Holm adjustment. In contrast, most item-format differences did not remain statistically significant after correction.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLM performance in licensing examinations was strongly influenced by domain, search access, and visual characteristics, whereas associations with response format were less consistent and model-dependent. These findings may inform assessment design and AI literacy education by clarifying how domain, visual demands, search access, and response structure influence LLM performance. Future assessments should prioritize construct-relevant professional competencies and evaluate both independent reasoning and the critical, ethical use of AI.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/23821205261483568","type":"journal-article","created":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T17:58:56Z","timestamp":1788803936000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Assessment Design in the Era of Large Language Models: Evidence From Japanese Health Professions Licensing Examinations"],"prefix":"10.1177","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-0628-3409","authenticated-orcid":false,"given":"Toshitsugu","family":"Sakurai","sequence":"first","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daichi","family":"Aizawa","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kazuyoshi","family":"Okawa","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryo","family":"Kofuchi","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masatsugu","family":"Hirota","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hiroya","family":"Gotouda","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Takatsugu","family":"Yamamoto","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chikahiro","family":"Ohkubo","sequence":"additional","affiliation":[{"name":"Tsurumi University School of Dental Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,9,7]]},"reference":[{"key":"e_1_3_6_2_2","doi-asserted-by":"publisher","DOI":"10.1097\/JS9.0000000000003668"},{"key":"e_1_3_6_3_2","doi-asserted-by":"publisher","DOI":"10.1097\/JS9.0000000000004413"},{"key":"e_1_3_6_4_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"e_1_3_6_5_2","doi-asserted-by":"publisher","DOI":"10.2196\/45312"},{"key":"e_1_3_6_6_2","doi-asserted-by":"publisher","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E. Capabilities of GPT-4 on medical challenge problems. 2023. arXiv. doi:10.48550\/arXiv.2303.13375.","DOI":"10.48550\/arXiv.2303.13375"},{"key":"e_1_3_6_7_2","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1166014"},{"key":"e_1_3_6_8_2","doi-asserted-by":"publisher","DOI":"10.3390\/info16070549"},{"key":"e_1_3_6_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tate.2006.12.021"},{"key":"e_1_3_6_10_2","doi-asserted-by":"publisher","DOI":"10.1080\/01421590701729930"},{"key":"e_1_3_6_11_2","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-200205000-00003"},{"key":"e_1_3_6_12_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1001_4"},{"key":"e_1_3_6_13_2","unstructured":"Ministry of Health Labour and Welfare. Reiwa 5-nenban shikaishi kokka shiken shutsudai kijun [2023 edition of the national examination blueprint for dentists]. Japanese. https:\/\/www.mhlw.go.jp\/stf\/newpage_24911.html. Accessed 16 Mar 2026."},{"key":"e_1_3_6_14_2","unstructured":"Ministry of Health Labour and Welfare. Atarashii yakuzaishi kokka shiken shutsudai kijun o sakutei shimashita [A new blueprint for the national examination for pharmacists was established]. Japanese. https:\/\/www.mhlw.go.jp\/stf\/shingi2\/0000143748.html. Accessed 16 Mar 2026."},{"key":"e_1_3_6_15_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0502_2"},{"key":"e_1_3_6_16_2","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2024.2342545"},{"key":"e_1_3_6_17_2","doi-asserted-by":"publisher","DOI":"10.1177\/20552076251346571"},{"key":"e_1_3_6_18_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-05944-8"},{"key":"e_1_3_6_19_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.11401"},{"key":"e_1_3_6_20_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2112.04426"},{"key":"e_1_3_6_21_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-05881-4"},{"key":"e_1_3_6_22_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.00020"},{"key":"e_1_3_6_23_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1908.02265"},{"key":"e_1_3_6_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2025.103265"},{"key":"e_1_3_6_25_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2929.2006.02578.x"},{"key":"e_1_3_6_26_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.09110"},{"key":"e_1_3_6_27_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0300-7"},{"key":"e_1_3_6_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.lindif.2023.102274"},{"key":"e_1_3_6_29_2","doi-asserted-by":"publisher","unstructured":"Bommasani R Hudson DA Adeli E et al. On the opportunities and risks of foundation models. 2021. arXiv. doi:10.48550\/arXiv.2108.07258.","DOI":"10.48550\/arXiv.2108.07258"},{"key":"e_1_3_6_30_2","unstructured":"Ministry of Health Labour and Welfare. Official answer key for the 120th National Medical Practitioners Qualifying Examination. Japanese. https:\/\/www.mhlw.go.jp\/general\/sikaku\/successlist\/2026\/siken01\/dl\/seitou.pdf. Accessed 16 Mar 2026."},{"key":"e_1_3_6_31_2","unstructured":"Ministry of Health Labour and Welfare. Official answer key for the 119th National Dental Practitioner Examination. Japanese. https:\/\/www.mhlw.go.jp\/general\/sikaku\/successlist\/2026\/siken02\/dl\/seitou.pdf. Accessed 25 Mar 2026."},{"key":"e_1_3_6_32_2","unstructured":"Ministry of Health Labour and Welfare. Pass criteria and official answers for the 111th National Pharmacist Examination. Japanese. https:\/\/www.mhlw.go.jp\/content\/11121000\/001678033.pdf. Accessed 25 Mar 2026."},{"key":"e_1_3_6_33_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.14165"},{"key":"e_1_3_6_34_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.11903"},{"key":"e_1_3_6_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF02295996"},{"key":"e_1_3_6_36_2","doi-asserted-by":"publisher","DOI":"10.1080\/14786440009463897"},{"key":"e_1_3_6_37_2","first-page":"31210","article-title":"Large language models can be easily distracted by irrelevant context","volume":"202","author":"Shi F","year":"2023","unstructured":"Shi F, Chen X, Misra K, et al. Large language models can be easily distracted by irrelevant context. Proceedings of the 40th International Conference on Machine Learning. 2023;202:31210-31227.","journal-title":"Proceedings of the 40th International Conference on Machine Learning"},{"key":"e_1_3_6_38_2","volume-title":"Proceedings of the Twelfth International Conference on Learning Representations","author":"Yoran O","year":"2024","unstructured":"Yoran O, Wolfson T, Ram O, Berant J. Making retrieval-augmented language models robust to irrelevant context. In: Proceedings of the Twelfth International Conference on Learning Representations; 2024."},{"key":"e_1_3_6_39_2","unstructured":"Ministry of Health Labour and Welfare. Announcement of the results of the 120th National Medical Practitioners Qualifying Examination. Japanese. https:\/\/www.mhlw.go.jp\/general\/sikaku\/successlist\/2026\/siken01\/about.html. Accessed 16 Mar 2026."},{"key":"e_1_3_6_40_2","unstructured":"Ministry of Health Labour and Welfare. Announcement of the results of the 119th National Dental Practitioner Examination. Japanese. https:\/\/www.mhlw.go.jp\/general\/sikaku\/successlist\/2026\/siken02\/about.html. Accessed 16 Mar 2026."},{"key":"e_1_3_6_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_3_6_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3571730"}],"container-title":["Journal of Medical Education and Curricular Development"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/23821205261483568","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/23821205261483568","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/23821205261483568","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T11:20:41Z","timestamp":1789125641000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/23821205261483568"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9]]},"references-count":41,"alternative-id":["10.1177\/23821205261483568"],"URL":"https:\/\/doi.org\/10.1177\/23821205261483568","relation":{},"ISSN":["2382-1205","2382-1205"],"issn-type":[{"value":"2382-1205","type":"print"},{"value":"2382-1205","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9]]},"article-number":"23821205261483568"}}