{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T02:47:59Z","timestamp":1790390879281,"version":"4.1.0"},"reference-count":26,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Recent studies have demonstrated that large language models exhibit exceptional performance in medical examinations. However, there is a lack of reports assessing their capabilities in specific domains or their application in practical data analysis using code interpreters. Furthermore, comparative analyses across different large language models have not been extensively conducted.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The purpose of this study was to evaluate whether advanced artificial intelligence (AI) models can analyze data from template-based input and demonstrate basic knowledge of reproductive medicine. Four AI models (GPT-4, GPT-4o, Claude 3.5 Sonnet, and Gemini Pro 1.5) were evaluated for their data analytical capabilities through numerical calculations and graph rendering. Their knowledge of infertility treatment was assessed using 10 examination questions developed by experts.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n First, we uploaded data to the AI models and furnished instruction templates using the chat interface. The study investigated whether the AI models could perform pregnancy rate analysis and graph rendering, based on blastocyst grades according to Gardner criteria. Second, we assessed model diagnostic capabilities based on specialized knowledge. This evaluation used 10 questions derived from the Japanese Fertility Specialist Examination and the Embryologist Certification Exam, along with chromosome imaging. These materials were curated under the supervision of certified embryologists and fertility specialists. All procedures were repeated 10 times per AI model.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n GPT-4o achieved grade A output (defined as achieving the objective with a single output attempt) in 9 out of 10 trials, outperforming GPT-4, which achieved grade A in 7 out of 10. The average processing times for data analysis were 26.8 (SD 3.7) seconds for GPT-4o and 36.7 (SD 3) seconds for GPT-4, whereas Claude failed in all 10 attempts. Gemini achieved an average processing time of 23 (SD 3) seconds and received grade A in 6 out of 10 trials, though occasional manual corrections were needed. Embryologists required an average of 358.3 (SD 9.7) seconds for the same tasks. In the knowledge-based assessment, GPT-4o, Claude, and Gemini achieved perfect scores (9\/9) on multiple-choice questions, while GPT-4 showed a 60% (6\/10) success rate on 1 question. None of the AI models could reliably diagnose chromosomal abnormalities from karyotype images, with the highest image diagnostic accuracy being 70% (7\/10) for Claude and Gemini.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This rapid processing demonstrates the potential for these AI models to significantly expedite data-intensive tasks in clinical settings. This performance underscores their potential utility as educational tools or decision support systems in reproductive medicine. However, none of the models were able to accurately interpret and diagnose using medical images.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/70107","type":"journal-article","created":{"date-parts":[[2025,8,13]],"date-time":"2025-08-13T06:55:09Z","timestamp":1755068109000},"page":"e70107-e70107","source":"Crossref","is-referenced-by-count":3,"title":["Application of Large Language Models in Data Analysis and Medical Education for Assisted Reproductive Technology: Comparative Study"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7347-5312","authenticated-orcid":false,"given":"Noriyuki","family":"Okuyama","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9009-915X","authenticated-orcid":false,"given":"Mika","family":"Ishii","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0777-0559","authenticated-orcid":false,"given":"Yuriko","family":"Fukuoka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0522-7760","authenticated-orcid":false,"given":"Hiromitsu","family":"Hattori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6369-1086","authenticated-orcid":false,"given":"Yuta","family":"Kasahara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4069-3724","authenticated-orcid":false,"given":"Tai","family":"Toshihiro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0839-0981","authenticated-orcid":false,"given":"Koki","family":"Yoshinaga","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6178-7204","authenticated-orcid":false,"given":"Tomoko","family":"Hashimoto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5211-7778","authenticated-orcid":false,"given":"Koichi","family":"Kyono","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,10,1]]},"reference":[{"key":"R1","doi-asserted-by":"crossref","unstructured":"Miyazaki K Sato R . Analyses of the technological accumulation over the 2nd and the 3rd AI boom and the issues related to AI adoption by firms. Presented at: 2018 Portland International Conference on Management of Engineering and Technology (PICMET); Aug 19-23, 2018:1-7; Honolulu, HI. [doi: 10.23919\/PICMET.2018.8481822]","DOI":"10.23919\/PICMET.2018.8481822"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1080\/07366981.2020.1847813","article-title":"What is digital transformation?","volume":"64","author":"Tang","journal-title":"EDPACS"},{"issue":"4","key":"R3","doi-asserted-by":"publisher","first-page":"124","DOI":"10.47832\/2757-5403.4-3.11","article-title":"The digital transformation revolution","volume":"3","author":"Abd-rabo","journal-title":"Int J Humanit Educ Res"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1177\/14604582221075554","article-title":"Digital transformation in medical education: factors that influence readiness","volume":"28","author":"Althubaiti","journal-title":"Health Informatics J"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.7189\/jogh.13.01003","article-title":"Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis","volume":"13","author":"Macdonald","journal-title":"J Glob Health"},{"issue":"Supplement_1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1093\/humrep\/dead093.103","article-title":"O-089\u2003Using ChatGPT to answer patient questions about fertility: the quality of information generated by a deep learning language model","volume":"38","author":"Beilby","journal-title":"Hum Reprod"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.2196\/46885","article-title":"The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers","volume":"9","author":"Eysenbach","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1002\/rmb2.12443","article-title":"A novel system based on artificial intelligence for predicting blastocyst viability and visualizing the explanation","volume":"21","author":"Enatsu","journal-title":"Reprod Med Biol"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.rbmo.2023.103308","article-title":"Improved pregnancy prediction performance in an updated deep-learning embryo selection model: a retrospective independent validation study","volume":"48","author":"Ueno","journal-title":"Reprod Biomed Online"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1002\/rmb2.12590","article-title":"Potential for artificial intelligence in medicine and its application to male infertility","volume":"23","author":"Kobayashi","journal-title":"Reprod Med Biol"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","DOI":"10.1016\/j.rbmo.2023.103735","article-title":"Identifying optimal puncture position by a real-time image analysis for Piezo-ICSI: a prospective randomized sibling oocyte study","volume":"48","author":"Morimoto","journal-title":"Reprod Biomed Online"},{"issue":"21","key":"R12","doi-asserted-by":"publisher","DOI":"10.3390\/jcm11216426","article-title":"Data-driven medicine in the diagnosis and treatment of infertility","volume":"11","author":"de Santiago","journal-title":"J Clin Med"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","DOI":"10.1093\/hropen\/hox011","article-title":"The Vienna consensus: report of an expert meeting on the development of art laboratory performance indicators","volume":"2017","author":"ESHRE Special Interest Group of Embryology","journal-title":"Human Reproduction Open"},{"key":"R14","unstructured":"Schulhoff S Ilie M Balepur N et al. The prompt report: a systematic survey of prompt engineering techniques. arXiv. Preprint posted online on Jun 6, 2024. [doi: 10.48550\/arXiv.2406.06608]"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Renze M Guven E . Self-reflection in large language model agents: effects on problem-solving performance. Presented at: 2024 2nd International Conference on Foundation and Large Language Models (FLLM); Nov 26-29, 2024:516-525; Dubai, United Arab Emirates. [doi: 10.1109\/FLLM63129.2024.10852426]","DOI":"10.1109\/FLLM63129.2024.10852426"},{"issue":"10","key":"R16","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1007\/s42452-024-06194-5","article-title":"How GPT models perform on the United States medical licensing examination: a systematic review","volume":"6","author":"Brin","journal-title":"Discov Appl Sci"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2023.1240915","article-title":"Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment","volume":"10","author":"Lai","journal-title":"Front Med (Lausanne)"},{"key":"R18","unstructured":"Kasai J Kasai Y Sakaguchi S Yamada Y Radev D . Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing examinations. arXiv. Preprint posted online on Mar 31, 2023. [doi: 10.48550\/arXiv.2303.18027]"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","journal-title":"Commun Med (Lond)"},{"issue":"11","key":"R20","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics12112771","article-title":"Diagnosis of cervical cancer and pre-cancerous lesions by artificial intelligence: a systematic review","volume":"12","author":"Allahqoli","journal-title":"Diagnostics (Basel)"},{"issue":"12","key":"R21","doi-asserted-by":"publisher","DOI":"10.1093\/dote\/doad048","article-title":"The role of artificial intelligence in the endoscopic diagnosis of esophageal cancer: a systematic review and meta-analysis","volume":"36","author":"Guidozzi","journal-title":"Dis Esophagus"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"154","DOI":"10.3390\/app15010154","article-title":"EduDCM: a novel framework for automatic educational dialogue classification dataset construction via distant supervision and large language models","volume":"15","author":"Qi","journal-title":"Appl Sci (Basel)"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3420709","article-title":"MCQGen: a large language model-driven MCQ generator for personalized learning","volume":"12","author":"Hang","journal-title":"IEEE Access"},{"issue":"3","key":"R24","doi-asserted-by":"publisher","first-page":"e28","DOI":"10.58600\/eurjther2201","article-title":"Hello AI: is it time for a revolutionary change in the health professional education system?","volume":"30","author":"Naqvi","journal-title":"Eur J Ther"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s44401-024-00006-z","article-title":"Preparing healthcare education for an AI-augmented future","volume":"1","author":"Zhang","journal-title":"Npj Health Syst"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250013","article-title":"Learning to teach AI: understanding the needs of healthcare professionals","volume":"322","author":"Jeyakumar","journal-title":"Stud Health Technol Inform"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e70107","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:58:31Z","timestamp":1782842311000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e70107"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,1]]},"references-count":26,"URL":"https:\/\/doi.org\/10.2196\/70107","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,1]]},"article-number":"v9i1e70107"}}