{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T08:00:33Z","timestamp":1790236833666,"version":"4.1.0"},"reference-count":34,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Cloud-hosted large vision-language models (LVLMs) often outperform open-weight models on multimodal benchmarks, but their applicability to health care examinations that require both text and image reasoning remains unclear. Existing studies on Japan\u2019s National Examination for Clinical Laboratory Technicians mainly used text-only settings and a limited set of models.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the accuracy and applicability of cloud-hosted LVLMs on this examination, quantify the contribution of images, examine alignment with human performance across medical subfields, and test generalizability on questions administered after the models\u2019 knowledge cutoff dates. We aim to provide a standardized benchmark of baseline multimodal performance under deployment-realistic conditions without additional fine-tuning.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n For this comparative benchmarking study, we built the Kensagishi question-answer dataset (Kensagishi QA), a benchmark of 800 questions from the examination (2020\u20102023), including text-only and image-based questions, in a standardized structured JSON format, and strict scoring criteria (all correct options required). We tested GPT-5, GPT-4o, GPT-4o-mini, Gemini 1.5 Pro, Gemini 2.5 Pro, and Neva-22B using a uniform \u201cnumbers-only\u201d response prompt; images were provided as Base64-encoded data when available. Accuracy was computed overall, by item type, and by medical subfield. Human reference was drawn from published examination results. Spearman correlation assessed model-human alignment. Generalization was evaluated on 200 questions from the 71st examination (2025), released after the models\u2019 knowledge cutoffs.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Top-performing models exceeded the 60% passing threshold. GPT-5 achieved 93.9% (95% CI 92.0%\u201095.4%) accuracy with images and 88.2% (95% CI 85.7%\u201090.2%) without images, Gemini 2.5 Pro 92.5% (95% CI 90.5%\u201094.1%) and 86.8% (95% CI 84.2%\u201088.9%), and GPT-4o 81.7% (95% CI 78.8%\u201084.2%) and 79.8% (95% CI 76.8%\u201082.4%), respectively. Providing images markedly improved accuracy for GPT-5, Gemini 2.5 Pro, and GPT-4o, whereas Neva-22B showed no improvement. Model-human rank correlations by medical subfield were moderate for GPT-4o (Spearman \u03c1=0.61; 95% CI \u20130.03 to 0.90;\n P<\/jats:italic>\n =.06 with images) and statistically significant for Gemini 1.5 Pro without images (\u03c1=0.67; 95% CI 0.07\u20100.91;\n P<\/jats:italic>\n =.03). On 200 postknowledge cutoff questions from the 71st examination (2025), model accuracies were comparable to historical averages (eg, GPT-5 91.5%, 95% CI 86.8%\u201094.6%; Gemini 2.5 Pro 90.7%, 95% CI 85.8%\u201094.0%; GPT-4o 82.0%, 95% CI 76.1%\u201086.7%), indicating no material performance degradation on unseen questions.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Cloud-hosted LVLMs show strong performance on this examination, and visual input is a key driver of accuracy for leading models. Models that better mirror human difficulty patterns (eg, GPT-4o) offer complementary insights for educational analytics, whereas the highest-accuracy models (eg, GPT-5 and Gemini 2.5 Pro) may be preferable when correctness is paramount. To our knowledge, this is the first systematic, multimodel evaluation isolating visual input that extends prior text-only, few-model studies and provides Kensagishi QA to guide educational deployment of medical AI.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/84266","type":"journal-article","created":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T07:10:07Z","timestamp":1784790607000},"page":"e84266-e84266","source":"Crossref","is-referenced-by-count":0,"title":["Performance of Cloud-Hosted Large Vision-Language Models on the Japanese National Examination for Clinical Laboratory Technicians: Comparative Benchmarking Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-4601-3023","authenticated-orcid":false,"given":"Kota","family":"Murakami","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2999-4734","authenticated-orcid":false,"given":"Ryosuke","family":"Matsuzawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8064-9595","authenticated-orcid":false,"given":"Yuya","family":"Tahara-Arai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1606-2762","authenticated-orcid":false,"given":"Haruka","family":"Ozaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,11]]},"reference":[{"key":"R1","unstructured":"Brown TB Mann B Ryder N Subbiah M Kaplan J Dhariwal P et al. Language models are few-shot learners. arXiv. Preprint posted online on Jul 22, 2020. [doi: 10.48550\/arXiv.2005.14165]"},{"key":"R2","unstructured":"OpenAIAdler S Agarwal S Ahmad L Akkaya I et al. GPT-4 technical report. arXiv. Preprint posted online on Mar 15, 2023. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3482107","article-title":"Survey of different large language model architectures: trends, benchmarks, and challenges","volume":"12","author":"Shao","journal-title":"IEEE Access"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Zhang D Yu Y Dong J et al. MM-LLMs: recent advances in multimodal large language models. Presented at: Findings of the Association for Computational Linguistics ACL 2024; Aug 11-16, 2024:12401-12430; Bangkok, Thailand and virtual meeting. 2024. URL: https:\/\/aclanthology.org\/2024.findings-acl.738\/ [Accessed 13-08-2026] [doi: 10.18653\/v1\/2024.findings-acl.738]","DOI":"10.18653\/v1\/2024.findings-acl.738"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Chou C Dunlap L Mashita K et al. VisionArena: 230K real world user-VLM conversations with preference labels. Presented at: 2025 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Jun 11-15, 2025:3877-3887; Nashville, TN, USA. 2025. [doi: 10.1109\/CVPR52734.2025.00367]","DOI":"10.1109\/CVPR52734.2025.00367"},{"key":"R6","doi-asserted-by":"crossref","unstructured":"Chen L Li J Dong X et al. Are we on the right way for evaluating large vision-language models? Presented at: Advances in Neural Information Processing Systems 37; Dec 10-15, 2024:27056-27087; Vancouver, BC, Canada. 2024. URL: http:\/\/www.proceedings.com\/79017.html [Accessed 13-08-2026] [doi: 10.52202\/079017-0850]","DOI":"10.52202\/079017-0850"},{"issue":"14","key":"R7","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? A large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","journal-title":"Appl Sci"},{"issue":"3","key":"R8","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","journal-title":"Nat Med"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Yang H Chen H Guo H et al. LLM-MedQA: enhancing medical question answering through case studies in large language models. Presented at: 2025 International Joint Conference on Neural Networks (IJCNN); Jun 30 to Jul 5, 2025; Rome, Italy. 2025. [doi: 10.1109\/IJCNN64981.2025.11228647]","DOI":"10.1109\/IJCNN64981.2025.11228647"},{"issue":"4","key":"R10","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.6359","article-title":"Semantic clinical artificial intelligence vs native large language model performance on the USMLE","volume":"8","author":"Elkin","journal-title":"JAMA Netw Open"},{"key":"R11","unstructured":"Kasai J Kasai Y Sakaguchi K Yamada Y Radev D . Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations. arXiv. Preprint posted online on Mar 31, 2023. [doi: 10.48550\/arXiv.2303.18027]"},{"key":"R12","doi-asserted-by":"crossref","unstructured":"Kido Y Yamada H Tokunaga T et al. Evaluation of LLM-generated distractors of multiple-choice questions for the Japanese National Nursing Examination. Presented at: 17th International Conference on Computer Supported Education; Apr 1-3, 2025:754-764; Porto, Portugal. 2025. URL: http:\/\/www.scitepress.org\/DigitalLibrary\/ProceedingLink.aspx?ID=1900 [Accessed 13-08-2026] [doi: 10.5220\/0013460300003932]","DOI":"10.5220\/0013460300003932"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"323","DOI":"10.14932\/jamt.23-80","article-title":"Performance of generative pretrained transformer on the national licensing examination for medical technologist in Japan","volume":"73","author":"Doi","journal-title":"Jpn J Med Technol"},{"key":"R14","volume":"16","author":"Ohshige","journal-title":"Jpn J Med Technol Educ"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000433","article-title":"Performance of generative pretrained transformer on the National Medical Licensing Examination in Japan","volume":"3","author":"Tanaka","journal-title":"PLoS Digit Health"},{"key":"R16","unstructured":"Questions and correct answers for the 66th national examination for medical technologists. Ministry of Health, Labour and Welfare. 2020. URL: https:\/\/www.mhlw.go.jp\/seisakunitsuite\/bunya\/kenkou_iryou\/iryou\/topics\/tp200414-07.html [Accessed 23-06-2025]"},{"key":"R17","unstructured":"Questions and correct answers for the 67th national examination for medical technologists. Ministry of Health, Labour and Welfare. 2021. URL: https:\/\/www.mhlw.go.jp\/seisakunitsuite\/bunya\/kenkou_iryou\/iryou\/topics\/tp210416-07.html [Accessed 23-06-2025]"},{"key":"R18","unstructured":"Questions and correct answers for the 68th national examination for medical technologists. Ministry of Health, Labour and Welfare. 2022. URL: https:\/\/www.mhlw.go.jp\/seisakunitsuite\/bunya\/kenkou_iryou\/iryou\/topics\/tp220421-07.html [Accessed 23-06-2025]"},{"key":"R19","unstructured":"Questions and correct answers for the 69th national examination for medical technologists. Ministry of Health, Labour and Welfare. 2023. URL: https:\/\/www.mhlw.go.jp\/seisakunitsuite\/bunya\/kenkou_iryou\/iryou\/topics\/tp230524-07.html [Accessed 23-06-2025]"},{"key":"R20","volume":"16","author":"Onodera","journal-title":"Jpn J Med Technol Educ"},{"issue":"158","key":"R21","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1080\/01621459.1927.10502953","article-title":"Probable inference, the law of succession, and statistical inference","volume":"22","author":"Wilson","journal-title":"J Am Stat Assoc"},{"issue":"8","key":"R22","doi-asserted-by":"publisher","first-page":"873","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<873::AID-SIM779>3.0.CO;2-I","article-title":"Interval estimation for the difference between independent proportions: Comparison of eleven methods","volume":"17","author":"Newcombe","journal-title":"Stat Med"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Murakami K Matsuzawa R Tahara-Arai Y Ozaki H . KensagishiQA: Benchmark data for evaluating large language models on the Japanese National Examination for Clinical Laboratory Technicians. GitHub. Jul 15, 2026. URL: https:\/\/github.com\/bioinfo-tsukuba\/KensagishiQA [Accessed 02-09-2026]","DOI":"10.2196\/preprints.84266"},{"key":"R24","unstructured":"Questions and correct answers for the 71st national examination for medical technologists. Ministry of Health, Labour and Welfare. 2025. URL: https:\/\/www.mhlw.go.jp\/seisakunitsuite\/bunya\/kenkou_iryou\/iryou\/topics\/tp250428-07.html [Accessed 17-07-2025]"},{"key":"R25","unstructured":"Ethical guidelines for medical and biological research involving human subjects. Ministry of Education, Culture, Sports, Science and Technology, Ministry of Health, Labour and Welfare, Ministry of Economy, Trade and Industry. 2023. URL: https:\/\/www.mhlw.go.jp\/content\/001457376.pdf [Accessed 18-07-2026]"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Li C Wong C Zhang S et al. LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. Presented at: Advances in Neural Information Processing Systems 36; Dec 10-16, 2023; New Orleans, Louisiana. 2023. URL: http:\/\/www.proceedings.com\/75280.html [Accessed 13-08-2026] [doi: 10.52202\/075280-1240]","DOI":"10.52202\/075280-1240"},{"key":"R27","unstructured":"Moor M Huang Q Wu S Yasunaga M Zakka C Dalmia Y et al. Med-Flamingo: a multimodal medical few-shot learner. arXiv. Preprint posted online on Jul 27, 2023. [doi: 10.48550\/arXiv.2307.15189]"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","journal-title":"NPJ Digit Med"},{"issue":"7972","key":"R29","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"key":"R30","unstructured":"Liu J Li Z Niu Q Zhang Z Xun Y Hou W et al. JMedEthicBench: a multi-turn conversational benchmark for evaluating medical safety in Japanese large language models. arXiv. Preprint posted online on Jan 4, 2026. [doi: 10.48550\/arXiv.2601.01627]"},{"key":"R31","unstructured":"Jiang J Huang J Aizawa A . JMedBench: a benchmark for evaluating Japanese biomedical large language models. In: Rambow O Wanner L Apidianaki M Al-Khalifa H Eugenio BD Schockaert S , editors. Presented at: Proceedings of the 31st international conference on computational linguistics; Jan 19-24, 2025:5918-5935; Abu Dhabi, UAE. 2025. URL: https:\/\/aclanthology.org\/2025.coling-main.395\/ [Accessed 31-08-2026]"},{"key":"R32","doi-asserted-by":"crossref","unstructured":"Wei J Wang X Schuurmans D et al. Chain-of-thought prompting elicits reasoning in large language models. Presented at: Advances in Neural Information Processing Systems 35; Dec 6-9, 2022:24824-24837; New Orleans, Louisiana. 2022. URL: http:\/\/www.proceedings.com\/68431.html [Accessed 13-08-2026] [doi: 10.52202\/068431-1800]","DOI":"10.52202\/068431-1800"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs","volume":"7","author":"Wang","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-06862-z","article-title":"GPT-4 generated answer rationales to multiple choice assessment questions in undergraduate medical education","volume":"25","author":"Ch\u2019en","journal-title":"BMC Med Educ"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T07:32:11Z","timestamp":1790235131000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e84266"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,11]]},"references-count":34,"URL":"https:\/\/doi.org\/10.2196\/84266","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,11]]},"article-number":"v12i14e84266"}}