{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T14:20:53Z","timestamp":1791210053585,"version":"4.1.0"},"reference-count":28,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T00:00:00Z","timestamp":1772409600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>To evaluate and compare the performance of nine contemporary LLM configurations on sleep medicine certification examination-aligned questions, analyzing version differences, pricing tiers, and subdomain competencies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Cross-sectional comparative analysis of 197 multiple-choice questions structured according to American Academy of Sleep Medicine (AASM) certification standards. Nine LLM configurations were evaluated: ChatGPT (GPT-3.5 free, GPT-4o paid), Gemini (2.5 Flash free, 2.5 Pro paid), Claude (3.7 Sonnet previous, Opus 4 paid), Deepseek V3 (free), xAI Grok3 (free), and Llama 3 (free). Each question was posed three times in independent sessions to minimize response variance. The first complete response from each iteration was recorded, and final accuracy was determined using strict 3\/3 concordance criterion (correct only when all three iterations yielded identical correct answers). While alternative scoring approaches exist (single-try accuracy, 2\/3 majority voting), the strict concordance method was selected as primary metric to minimize stochastic variation and ensure robust performance estimates. Supplementary analyses using majority voting (2\/3) yielded consistent model rankings with marginally higher absolute accuracy values. Performance metrics included overall accuracy rates, 95% confidence intervals, and subdomain-specific analyses across seven sleep medicine categories. Statistical analyses employed Pearson\u2019s chi-square test for heterogeneity and McNemar\u2019s test for pairwise comparisons. This text-based simulation evaluated model performance on certification-style questions, though it does not replicate actual clinical examination conditions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Model performance demonstrated significant heterogeneity (\n                      <jats:italic>\u03c7<\/jats:italic>\n                      <jats:sup>2<\/jats:sup>\n                      \u202f=\u202f101.95, df\u202f=\u202f8,\n                      <jats:italic>p<\/jats:italic>\n                      \u202f&amp;lt;\u202f0.001), with accuracy rates ranging from 68.5% to 95.9%. Gemini 2.5 Pro achieved the highest overall accuracy (95.9%, 95% CI: 93.2\u201398.7%), followed by Claude Opus 4 (93.9%, 95% CI: 90.6\u201397.2%) and ChatGPT GPT-4o (93.4%, 95% CI: 89.9\u201396.9%). Premium versions consistently demonstrated superior performance compared to free alternatives, with performance differences ranging from 5.1 to 8.6 points (all\n                      <jats:italic>p<\/jats:italic>\n                      \u202f&amp;lt;\u202f0.05). Subdomain analysis revealed the highest performance consistency in Secondary Sleep Disorders (92.0% mean accuracy) and the greatest variability in Diagnostic Methods (85.9% mean accuracy). Sensitivity analysis comparing three scoring criteria (single-try \u22651\/3, majority voting \u22652\/3, strict concordance 3\/3) revealed that scoring methodology had minimal impact on model rankings (Spearman\u2019s\n                      <jats:italic>\u03c1<\/jats:italic>\n                      \u202f=\u202f0.879\u20131.000, all\n                      <jats:italic>p<\/jats:italic>\n                      \u202f&amp;lt;\u202f0.01). Majority voting and strict concordance yielded identical accuracy rates in seven of nine models due to high response consistency (95.8% average). Eight of nine models exceeded the 80% reference benchmark under all three scoring criteria.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Contemporary LLMs demonstrate substantially improved performance compared to previous evaluations, with premium models exceeding the 80% reference benchmark. However, these results reflect performance on a certification-aligned question bank rather than the official board examination itself. The significant performance advantage of paid versions raises important considerations regarding equitable access to AI-enhanced medical education and clinical decision support tools.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fmed.2026.1761025","type":"journal-article","created":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T06:42:37Z","timestamp":1772433757000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Performance of large language models on sleep medicine certification examination: a comprehensive multi-model analysis"],"prefix":"10.3389","volume":"13","author":[{"given":"Abdurrahman","family":"Ko\u00e7","sequence":"first","affiliation":[{"name":"Department of Pulmonary Medicine, Meram State Hospital","place":["Konya, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abdullah Enes","family":"Ata\u015f","sequence":"additional","affiliation":[{"name":"Department of Radiology, Necmettin Erbakan University","place":["Konya, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"\u015eebnem","family":"Yosunkaya","sequence":"additional","affiliation":[{"name":"Department of Pulmonary Medicine, Necmettin Erbakan University","place":["Konya, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"H\u00fclya","family":"Vatansev","sequence":"additional","affiliation":[{"name":"Department of Pulmonary Medicine, Necmettin Erbakan University","place":["Konya, T\u00fcrkiye"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,3,2]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesk\u00f3","year":"2023","journal-title":"NPJ Digit Med"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLoS Digit Health"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1093\/ehjdh\/ztad029","article-title":"ChatGPT takes on the European exam in core cardiology: an artificial intelligence success story?","volume":"4","author":"Skalidis","year":"2023","journal-title":"Eur Heart J Digit Health"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"e230987","DOI":"10.1148\/radiol.230987","article-title":"GPT-4 in radiology: improvements in advanced reasoning","volume":"307","author":"Bhayana","year":"2023","journal-title":"Radiology"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"4271","DOI":"10.1007\/s00405-023-08051-4","article-title":"Chatgpt's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions","volume":"280","author":"Hoch","year":"2023","journal-title":"Eur Arch Otorrinolaringol"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"100324","DOI":"10.1016\/j.xops.2023.100324","article-title":"Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings","volume":"3","author":"Antaki","year":"2023","journal-title":"Ophthalmol Sci"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"1558","DOI":"10.1093\/jamia\/ocad104","article-title":"ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?","volume":"30","author":"Kumah-Crystal","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"221","DOI":"10.5664\/jcsm.1790","article-title":"Development and growth of a large multispecialty certification examination: sleep medicine certification\u2014results of the first three examinations","volume":"8","author":"Quan","year":"2012","journal-title":"J Clin Sleep Med"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/S2213-2600(19)30198-5","article-title":"Estimation of the global prevalence and burden of obstructive sleep apnoea: a literature-based analysis","volume":"7","author":"Benjafield","year":"2019","journal-title":"Lancet Respir Med"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"568","DOI":"10.1126\/science.abj8188","article-title":"The translational neuroscience of sleep: a contextual framework","volume":"374","author":"Grandner","year":"2021","journal-title":"Science"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"e18","DOI":"10.1161\/CIR.0000000000001078","article-title":"Life's essential 8: updating and enhancing the American Heart Association's construct of cardiovascular health: a presidential advisory from the American Heart Association","volume":"146","author":"Lloyd-Jones","year":"2022","journal-title":"Circulation"},{"key":"ref13","first-page":"42","article-title":"Obezite hipoventilasyon sendromlu kad\u0131n hastalar\u0131n de\u011ferlendirilmesi","volume":"28","author":"Yosunkaya","year":"2012","journal-title":"Selcuk Tip Derg"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"2137","DOI":"10.1007\/s00405-023-08381-3","article-title":"Performance of artificial intelligence chatbots in sleep medicine certification board exams: ChatGPT versus Google bard","volume":"281","author":"Cheong","year":"2024","journal-title":"Eur Arch Otorrinolaringol"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"1511","DOI":"10.5664\/jcsm.11732","article-title":"Artificial intelligence in sleep medicine: assessing the diagnostic precision of ChatGPT-4","volume":"21","author":"Patel","year":"2025","journal-title":"J Clin Sleep Med"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1631","DOI":"10.1007\/s00405-024-08985-3","article-title":"Chasing sleep physicians: ChatGPT-4o on the interpretation of polysomnographic results","volume":"282","author":"Seifen","year":"2025","journal-title":"Eur Arch Otorrinolaringol"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"e58375","DOI":"10.2196\/58375","article-title":"Performance of plug-in augmented ChatGPT and its ability to quantify uncertainty: simulation study on the German medical board examination","volume":"11","author":"Madrid","year":"2025","journal-title":"JMIR Med Educ"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"605","DOI":"10.12669\/pjms.39.2.7653","article-title":"ChatGPT\u2014reshaping medical education and clinical management","volume":"39","author":"Khan","year":"2023","journal-title":"Pak J Med Sci"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"e60807","DOI":"10.2196\/60807","article-title":"Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis","volume":"26","author":"Liu","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"e66114","DOI":"10.2196\/66114","article-title":"Large language model-based performance assessment of 198 medical licensing examinations in 28 countries: cross-national comparative study","volume":"26","author":"Zong","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1186\/s12909-024-05125-7","article-title":"Performance of ChatGPT on Chinese National Medical Licensing Examinations: a five-year examination evaluation study for physicians, pharmacists and nurses","volume":"24","author":"Zong","year":"2024","journal-title":"BMC Med Educ"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"913168","DOI":"10.3389\/frai.2022.913168","article-title":"The fourth industrial revolution\u2014smart technology, artificial intelligence, robotics and algorithms: industrial psychologists in future workplaces","volume":"5","author":"Oosthuizen","year":"2022","journal-title":"Front Artif Intell"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1080\/0142159X.2023.2186203","article-title":"Ethical use of artificial intelligence in health professions education: AMEE guide no. 158","volume":"45","author":"Masters","year":"2023","journal-title":"Med Teach"},{"key":"ref24","author":"Nori","year":"2023"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"E2712","DOI":"10.1002\/lary.29725","article-title":"Characteristics of the dual board-certified sleep otolaryngology workforce","volume":"131","author":"Yu","year":"2021","journal-title":"Laryngoscope"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"1423","DOI":"10.5664\/jcsm.9214","article-title":"Impact of obstructive sleep apnea on cardiometabolic health in a random sample of older adults in rural South Africa: building the case for the treatment of sleep disorders in underresourced settings","volume":"17","author":"Roche","year":"2021","journal-title":"J Clin Sleep Med"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/j.clinimag.2021.09.018","article-title":"Impact of artificial intelligence on US medical students' choice of radiology","volume":"81","author":"Reeder","year":"2022","journal-title":"Clin Imaging"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"e0000604","DOI":"10.1371\/journal.pdig.0000604","article-title":"Performance of publicly available large language models on internal medicine board-style questions","volume":"3","author":"Tarabanis","year":"2024","journal-title":"PLOS Digit Health"}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1761025\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T06:42:38Z","timestamp":1772433758000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2026.1761025\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,2]]},"references-count":28,"alternative-id":["10.3389\/fmed.2026.1761025"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2026.1761025","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,2]]},"article-number":"1761025"}}