{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T05:02:54Z","timestamp":1789448574510,"version":"build-2803163510"},"reference-count":25,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T00:00:00Z","timestamp":1789430400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:p>\n                    This study asked whether five contemporary large language models answer medical genetics multiple-choice questions with equivalent accuracy on published versus novel items and across English and Turkish, and sought to characterize the errors that persist. Five models (GPT-5.2, Gemini 3 Pro, Claude Sonnet 4.6, Grok 4, and DeepSeek-V3.2) answered 100 four-option questions (50 from a published board review; 50 novel, expert-authored items absent from any database) in English and Turkish, yielding 1,000 responses. Correctness was modeled with item-clustered generalized estimating equation and Bayesian mixed-effects logistic regression (the latter as a prespecified sensitivity analysis); question provenance and language were tested for equivalence (item-clustered two one-sided tests, \u00b15-percentage-point margin), and the paired language effect with the McNemar test. Inter-model agreement and error concordance were examined. Overall accuracy was 97.7%. In the item-clustered GEE, only Gemini 3 Pro nominally exceeded the lowest-performing model; this imprecise contrast did not remain significant after Holm correction for the four secondary model comparisons, whereas the Bayesian sensitivity analysis additionally yielded a credible interval excluding 1 for GPT-5.2 versus Grok 4. Accuracy was statistically equivalent within the prespecified margin for published versus novel items (difference, \u22121.4 percentage points; item-clustered 90% CI, \u22124.1 to +1.3; item-clustered TOST\n                    <jats:italic>P<\/jats:italic>\n                    \u2009=\u20090.014) and for English versus Turkish (difference, \u22120.6 points; paired item-level TOST\n                    <jats:italic>P<\/jats:italic>\n                    \u2009&amp;lt;\u20090.001; McNemar\n                    <jats:italic>P<\/jats:italic>\n                    \u2009=\u20090.58). Inter-model agreement on the exact option chosen was high (Fleiss\n                    <jats:italic>\u03ba<\/jats:italic>\n                    , 0.94\u20130.97). All five items with two or more errors failed concordantly, every erring model selecting the same wrong option. Beyond high accuracy, distinct model families converged on identical answers and, on the hardest items, on identical errors, a pattern consistent with shared learned associations and\/or overlapping training data, although this behavioral comparison cannot identify the underlying mechanism. In an exploratory analysis, residual errors showed concentrated and concordant patterns, clustering in multistep Bayesian reasoning and evolving facts, suggesting that a second model may provide limited independent protection on such items. Performance on this restricted multiple-choice benchmark does not establish readiness for clinical genetic counseling, variant interpretation, or quantitative risk assessment, and human oversight remains essential.\n                  <\/jats:p>","DOI":"10.3389\/fmed.2026.1938886","type":"journal-article","created":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T04:26:46Z","timestamp":1789446406000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking five large language models in medical genetics: a bilingual comparative evaluation using published and novel expert-authored questions"],"prefix":"10.3389","volume":"13","author":[{"given":"\u00d6zge Beyza","family":"G\u00fcndo\u011fdu \u00d6\u011f\u00fctl\u00fc","sequence":"first","affiliation":[{"name":"Department of Medical Genetics, Ankara Etlik City Hospital","place":["Ankara, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin D.","family":"Solomon","sequence":"additional","affiliation":[{"name":"Medical Genomics Unit, National Human Genome Research Institute, National Institutes of Health","place":["Bethesda, MD, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yusuf Selman","family":"\u00c7elik","sequence":"additional","affiliation":[{"name":"Department of Child and Adolescent Psychiatry, K\u0131r\u0131kkale University","place":["K\u0131r\u0131kkale, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,15]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"23821205241238641","DOI":"10.1177\/23821205241238641","article-title":"Can ChatGPT-3.5 pass a medical exam? A systematic review of ChatGPT\u2019s performance in academic testing","volume":"11","author":"Sumbal","year":"2024","journal-title":"J Med Educ Curric Dev"},{"key":"B2","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1186\/s12911-025-02954-4","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"Shool","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"key":"B3","doi-asserted-by":"publisher","first-page":"603","DOI":"10.3390\/healthcare13060603","article-title":"A review of large language models in medical education, clinical decision support, and healthcare administration","volume":"13","author":"Vrdoljak","year":"2025","journal-title":"Healthcare (Basel)"},{"key":"B4","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat Med"},{"key":"B5","author":"Hou","year":"2025"},{"key":"B6","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1186\/s12911-024-02709-7","article-title":"Analyzing evaluation methods for large language models in the medical field: a scoping review","volume":"24","author":"Lee","year":"2024","journal-title":"BMC Med Inform Decis Mak"},{"key":"B7","doi-asserted-by":"publisher","first-page":"e53","DOI":"10.12771\/emj.2025.00787","article-title":"Technologies, opportunities, challenges, and future directions for integrating generative artificial intelligence into medical education: a narrative review","volume":"48","author":"Kang","year":"2025","journal-title":"Ewha Med J"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1694070","DOI":"10.3389\/fgene.2025.1694070","article-title":"A systematic review on the generative AI applications in human medical genetics","volume":"16","author":"Changalidis","year":"2026","journal-title":"Front Genet"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/S1525-1578(10)60484-9","article-title":"Bayesian Analysis and risk assessment in genetic counseling and testing","volume":"6","author":"Ogino","year":"2004","journal-title":"J Mol Diagn"},{"key":"B10","doi-asserted-by":"publisher","first-page":"466","DOI":"10.1038\/s41431-023-01396-8","article-title":"Analysis of large-language model versus human performance for genetics questions","volume":"32","author":"Duong","year":"2024","journal-title":"Eur J Hum Genet"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1819","DOI":"10.1016\/j.ajhg.2024.07.011","article-title":"Evaluating large language models on medical, lay-language, and self-reported descriptions of genetic conditions","volume":"111","author":"Flaharty","year":"2024","journal-title":"Am J Hum Genet"},{"key":"B12","doi-asserted-by":"publisher","first-page":"2271","DOI":"10.1093\/jamia\/ocae128","article-title":"A comparative evaluation of ChatGPT 3.5 and ChatGPT 4 in responses to selected genetics questions","volume":"31","author":"McGrath","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"B13","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1038\/s41431-023-01419-4","article-title":"Can ChatGPT understand genetics?","volume":"32","author":"Emmert-Streib","year":"2024","journal-title":"Eur J Hum Genet"},{"key":"B14","doi-asserted-by":"publisher","first-page":"42096","DOI":"10.1038\/s41598-025-26036-7","article-title":"The pitfalls of multiple-choice questions in generative AI and medical education","volume":"15","author":"Singh","year":"2025","journal-title":"Sci Rep"},{"key":"B15","doi-asserted-by":"publisher","first-page":"e58898","DOI":"10.2196\/58898","article-title":"Factors associated with the accuracy of large language models in basic medical science examinations: cross-sectional study","volume":"11","author":"Kaewboonlert","year":"2025","journal-title":"JMIR Med Educ"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1002\/9781119847212","volume-title":"Medical Genetics and Genomics: Questions for Board Review","author":"Solomon","year":"2022"},{"key":"B17","doi-asserted-by":"publisher","first-page":"706","DOI":"10.1016\/j.jfo.2023.05.006","article-title":"Success of ChatGPT, an AI language model, in taking the French language version of the European board of ophthalmology examination: a novel approach to medical knowledge assessment","volume":"46","author":"Panthier","year":"2023","journal-title":"J Fr Ophtalmol"},{"key":"B18","doi-asserted-by":"publisher","first-page":"e0000397","DOI":"10.1371\/journal.pdig.0000397","article-title":"How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language","volume":"2","author":"Fang","year":"2023","journal-title":"PLOS Digit Health"},{"key":"B19","doi-asserted-by":"publisher","first-page":"105957","DOI":"10.1016\/j.ebiom.2025.105957","article-title":"Consistent performance of large language models in rare disease diagnosis across ten languages and 4917 cases","volume":"121","author":"Chimirri","year":"2025","journal-title":"EBioMedicine"},{"key":"B20","author":"Hendrycks","year":"2020"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11126-026-10275-6","article-title":"Artificial intelligence in psychiatry training: comparative insights from nine large language models across cultural and exam contexts","author":"\u00c7elik","year":"2026","journal-title":"Psychiatr Q"},{"key":"B22","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1111\/pcn.13588","article-title":"The now and future of ChatGPT and GPT in psychiatry","volume":"77","author":"Cheng","year":"2023","journal-title":"Psychiatry Clin Neurosci"},{"key":"B23","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1186\/s12909-024-05125-7","article-title":"Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses","volume":"24","author":"Zong","year":"2024","journal-title":"BMC Med Educ"},{"key":"B24","doi-asserted-by":"publisher","first-page":"106424","DOI":"10.1016\/j.ijmedinf.2026.106424","article-title":"Precision grounding: augmenting large language models with evidence-based databases for trustworthy genetic variant summarization","volume":"214","author":"Du","year":"2026","journal-title":"Int J Med Inform"},{"key":"B25","doi-asserted-by":"publisher","first-page":"15093","DOI":"10.1038\/s41598-025-99539-y","article-title":"The application of large language models to the phenotype-based prioritization of causative genes in rare disease patients","volume":"15","author":"Kafkas","year":"2025","journal-title":"Sci Rep"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1938886\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T04:26:47Z","timestamp":1789446407000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1938886\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,15]]},"references-count":25,"alternative-id":["10.3389\/fmed.2026.1938886"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1938886","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,15]]},"article-number":"1938886"}}