{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T19:19:45Z","timestamp":1791227985383,"version":"4.1.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T00:00:00Z","timestamp":1783123200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T00:00:00Z","timestamp":1790208000000},"content-version":"vor","delay-in-days":82,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Oral Health"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Pharmacology represents the lowest-performing subcategory in oral and maxillofacial surgery (OMFS) evaluations of large language models (LLMs), yet no study has simultaneously compared the leading commercial LLMs across multiple pharmacological domains and question formats. This study evaluated ChatGPT 5.3, Gemini 3.1 Pro, and Claude 4.6 Sonnet in OMFS pharmacology.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      Thirty-six OMFS pharmacology questions spanning five clinical domains (antibiotic prophylaxis, analgesics, drug\u2013drug interactions, anesthetic pharmacology, special populations) and three formats (open-ended, multiple-choice, true\/false;\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u200912 each) were submitted to each LLM using a standardized role-conditioning prompt. The 108 responses were independently and blindly evaluated by two oral and maxillofacial surgeons (one specialist and one resident) on three 5-point Likert criteria. Inter-rater reliability was quantified using ICC(2,1) and Cohen\u2019s \u03ba_w. Inter-model differences were assessed using Friedman tests; format effects were assessed using Kruskal\u2013Wallis tests with Bonferroni-corrected post-hoc comparisons.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Inter-rater reliability was excellent (ICC\u2009=\u20090.828; \u03ba_w\u2009=\u20090.827; exact agreement 91.0%). A robust hierarchy emerged: Claude\u2009&gt;\u2009Gemini\u2009&gt;\u2009ChatGPT (\u03c7\u00b2(2)\u2009=\u200947.91,\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001, W\u2009=\u20090.665), with all pairwise comparisons significant. Gemini and Claude did not differ significantly in any format section, indicating clinical equivalence. ChatGPT exhibited a significant decline on open-ended, integrative-reasoning items (H(2)\u2009=\u200917.04,\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001, \u03b5\u00b2 = 0.456), absent in Gemini and Claude. Significant positive correlations among the evaluation criteria within the ChatGPT data indicated convergence among the three scoring dimensions.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Claude 4.6 Sonnet and Gemini 3.1 Pro achieved near-maximal scores on this structured pharmacology benchmark, while ChatGPT 5.3 showed a significant decline in open-ended reasoning. Current LLMs should be regarded as adjunctive tools requiring expert verification for high-risk OMFS pharmacological decisions.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12903-026-09182-w","type":"journal-article","created":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T01:26:18Z","timestamp":1783128378000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Do large language models differ in their pharmacology-related response quality for oral and maxillofacial surgery? a blinded expert benchmark study"],"prefix":"10.1186","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3326-0047","authenticated-orcid":false,"given":"Mustafa","family":"Isleyen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8667-3933","authenticated-orcid":false,"given":"Asenur","family":"Aydemir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,7,4]]},"reference":[{"key":"9182_CR1","doi-asserted-by":"publisher","first-page":"e68560","DOI":"10.2196\/68560","volume":"27","author":"HS Yun","year":"2025","unstructured":"Yun HS, Bickmore T. Online Health Information-Seeking in the Era of Large Language Models: Cross-Sectional Web-Based Survey Study. J Med Internet Res. 2025;27:e68560. https:\/\/doi.org\/10.2196\/68560.","journal-title":"J Med Internet Res"},{"key":"9182_CR2","doi-asserted-by":"publisher","first-page":"60","DOI":"10.3390\/antibiotics14010060","volume":"14","author":"NI Antonie","year":"2025","unstructured":"Antonie NI, Gheorghe G, Ionescu VA, Tiuc\u0103 L-C, Diaconu CC. The Role of ChatGPT and AI Chatbots in Optimizing Antibiotic Therapy: A Comprehensive Narrative Review. Antibiot (Basel). 2025;14:60. https:\/\/doi.org\/10.3390\/antibiotics14010060.","journal-title":"Antibiot (Basel)"},{"key":"9182_CR3","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1016\/j.jdsr.2026.01.002","volume":"62","author":"KAPLAN TT, AKYOL GE","year":"2026","unstructured":"KAPLAN TT, AKYOL GE. The role of large language models in dental diagnosis, decision-making, and communication: A systematic review. Jpn Dent Sci Rev. 2026;62:57\u201367. https:\/\/doi.org\/10.1016\/j.jdsr.2026.01.002.","journal-title":"Jpn Dent Sci Rev"},{"key":"9182_CR4","doi-asserted-by":"publisher","first-page":"1363","DOI":"10.3390\/jcm14041363","volume":"14","author":"S-W On","year":"2025","unstructured":"On S-W, Cho S-W, Park S-Y, Ha J-W, Yi S-M, Park I-Y, et al. Chat Generative Pre-Trained Transformer (ChatGPT) in Oral and Maxillofacial Surgery: A Narrative Review on Its Research Applications and Limitations. J Clin Med. 2025;14:1363. https:\/\/doi.org\/10.3390\/jcm14041363.","journal-title":"J Clin Med"},{"key":"9182_CR5","doi-asserted-by":"publisher","first-page":"881","DOI":"10.1016\/j.ijom.2024.06.003","volume":"53","author":"B Quah","year":"2024","unstructured":"Quah B, Yong CW, Lai CWM, Islam I. Performance of large language models in oral and maxillofacial surgery examinations. Int J Oral Maxillofac Surg. 2024;53:881\u20136. https:\/\/doi.org\/10.1016\/j.ijom.2024.06.003.","journal-title":"Int J Oral Maxillofac Surg"},{"key":"9182_CR6","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/j.bjoms.2025.08.015","volume":"64","author":"R Grillo","year":"2026","unstructured":"Grillo R, Llanos AH, Costa C, Melhem-Elias F. Comparison of large language models in oral and maxillofacial surgery. Br J Oral Maxillofac Surg. 2026;64:43\u20139. https:\/\/doi.org\/10.1016\/j.bjoms.2025.08.015.","journal-title":"Br J Oral Maxillofac Surg"},{"key":"9182_CR7","doi-asserted-by":"publisher","first-page":"660","DOI":"10.1186\/s12903-026-08041-y","volume":"26","author":"L Liu","year":"2026","unstructured":"Liu L, Ma K, Wang Y. Performance of five large language models in oral and maxillofacial surgery exam questions: a comparative study. BMC Oral Health. 2026;26:660. https:\/\/doi.org\/10.1186\/s12903-026-08041-y.","journal-title":"BMC Oral Health"},{"key":"9182_CR8","doi-asserted-by":"publisher","first-page":"e963","DOI":"10.1161\/CIR.0000000000000969","volume":"143","author":"WR Wilson","year":"2021","unstructured":"Wilson WR, Gewitz M, Lockhart PB, Bolger AF, DeSimone DC, Kazi DS, et al. Prevention of Viridans Group Streptococcal Infective Endocarditis: A Scientific Statement From the American Heart Association. Circulation. 2021;143:e963\u201378. https:\/\/doi.org\/10.1161\/CIR.0000000000000969.","journal-title":"Circulation"},{"key":"9182_CR9","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1016\/j.identj.2024.09.033","volume":"75","author":"P Rewthamrongsris","year":"2025","unstructured":"Rewthamrongsris P, Burapacheep J, Trachoo V, Porntaveetus T. Accuracy of Large Language Models for Infective Endocarditis Prophylaxis in Dental Procedures. Int Dent J. 2025;75:206\u201312. https:\/\/doi.org\/10.1016\/j.identj.2024.09.033.","journal-title":"Int Dent J"},{"key":"9182_CR10","doi-asserted-by":"publisher","first-page":"1336","DOI":"10.1016\/j.cmi.2025.03.002","volume":"31","author":"A De Vito","year":"2025","unstructured":"De Vito A, Geremia N, Bavaro DF, Seo SK, Laracy J, Mazzitelli M, et al. Comparing large language models for antibiotic prescribing in different clinical scenarios: which performs better? Clin Microbiol Infect. 2025;31:1336\u201342. https:\/\/doi.org\/10.1016\/j.cmi.2025.03.002.","journal-title":"Clin Microbiol Infect"},{"key":"9182_CR11","doi-asserted-by":"publisher","DOI":"10.1186\/s12903-026-07996-2","author":"NY \u00c7\u0131rako\u011flu","year":"2026","unstructured":"\u00c7\u0131rako\u011flu NY, Do\u011fan M, Duymaz BC. Comparative evaluation of artificial intelligence language models on knowledge of dental antibiotic use. BMC Oral Health. 2026. https:\/\/doi.org\/10.1186\/s12903-026-07996-2.","journal-title":"BMC Oral Health"},{"key":"9182_CR12","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1016\/j.bjoms.2026.01.002","volume":"64","author":"BK Frutuoso Maia","year":"2026","unstructured":"Frutuoso Maia BK, de Morais EF, de Santana Santos T, Charles Pagotto LE. Performance of large language models in preoperative and postoperative counselling for aesthetic facial procedures. Br J Oral Maxillofac Surg. 2026;64:216\u201322. https:\/\/doi.org\/10.1016\/j.bjoms.2026.01.002.","journal-title":"Br J Oral Maxillofac Surg"},{"key":"9182_CR13","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1038\/s41746-024-01258-7","volume":"7","author":"TYC Tam","year":"2024","unstructured":"Tam TYC, Sivarajkumar S, Kapoor S, Stolyar AV, Polanska K, McCarthy KR, et al. A framework for human evaluation of large language models in healthcare derived from literature review. NPJ Digit Med. 2024;7:258. https:\/\/doi.org\/10.1038\/s41746-024-01258-7.","journal-title":"NPJ Digit Med"},{"key":"9182_CR14","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","volume":"15","author":"TK Koo","year":"2016","unstructured":"Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15:155\u201363. https:\/\/doi.org\/10.1016\/j.jcm.2016.02.012.","journal-title":"J Chiropr Med"},{"key":"9182_CR15","doi-asserted-by":"publisher","first-page":"e70302","DOI":"10.7759\/cureus.70302","volume":"16","author":"M Fujimoto","year":"2024","unstructured":"Fujimoto M, Kuroda H, Katayama T, Yamaguchi A, Katagiri N, Kagawa K, et al. Evaluating Large Language Models in Dental Anesthesiology: A Comparative Analysis of ChatGPT-4, Claude 3 Opus, and Gemini 1.0 on the Japanese Dental Society of Anesthesiology Board Certification Exam. Cureus. 2024;16:e70302. https:\/\/doi.org\/10.7759\/cureus.70302.","journal-title":"Cureus"},{"key":"9182_CR16","doi-asserted-by":"publisher","first-page":"e76925","DOI":"10.2196\/76925","volume":"11","author":"H Sato","year":"2025","unstructured":"Sato H, Ogasawara K, Sakurai H. Performance Evaluation of 18 Generative AI Models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese Pharmacist Licensing Examination: Comparative Study. JMIR Med Educ. 2025;11:e76925. https:\/\/doi.org\/10.2196\/76925.","journal-title":"JMIR Med Educ"},{"key":"9182_CR17","doi-asserted-by":"publisher","first-page":"e0317423","DOI":"10.1371\/journal.pone.0317423","volume":"20","author":"HC Nguyen","year":"2025","unstructured":"Nguyen HC, Dang HP, Nguyen TL, Hoang V, Nguyen VA. Accuracy of latest large language models in answering multiple choice questions in dentistry: A comparative study. PLoS ONE. 2025;20:e0317423. https:\/\/doi.org\/10.1371\/journal.pone.0317423.","journal-title":"PLoS ONE"},{"key":"9182_CR18","doi-asserted-by":"publisher","first-page":"33083","DOI":"10.1038\/s41598-025-17030-0","volume":"15","author":"D W\u00f3jcik","year":"2025","unstructured":"W\u00f3jcik D, Adamiak O, Czerepak G, Tokarczuk O, Szalewski L. Comparing the performance of ChatGPT, Gemini, and Claude in English and Polish on medical examinations. Sci Rep. 2025;15:33083. https:\/\/doi.org\/10.1038\/s41598-025-17030-0.","journal-title":"Sci Rep"},{"key":"9182_CR19","doi-asserted-by":"publisher","first-page":"1541976","DOI":"10.3389\/froh.2025.1541976","volume":"6","author":"K Ji","year":"2025","unstructured":"Ji K, Wu Z, Han J, Zhai G, Liu J. Evaluating ChatGPT-4\u2019s performance on oral and maxillofacial queries: Chain of Thought and standard method. Front Oral Health. 2025;6:1541976. https:\/\/doi.org\/10.3389\/froh.2025.1541976.","journal-title":"Front Oral Health"},{"key":"9182_CR20","doi-asserted-by":"publisher","first-page":"261","DOI":"10.3390\/dj13060261","volume":"13","author":"V Ronsivalle","year":"2025","unstructured":"Ronsivalle V, Santonocito S, Cammarata U, Lo Muzio E, Cicci\u00f9 M. Current Applications of Chatbots Powered by Large Language Models in Oral and Maxillofacial Surgery: A Systematic Review. Dent J (Basel). 2025;13:261. https:\/\/doi.org\/10.3390\/dj13060261.","journal-title":"Dent J (Basel)"},{"key":"9182_CR21","doi-asserted-by":"publisher","first-page":"102644","DOI":"10.1016\/j.jormas.2025.102644","volume":"127","author":"BE Yilmaz","year":"2026","unstructured":"Yilmaz BE, Ozbey F, Gokkurt Yilmaz BN, Akpinar H. Can large language models perform clinical anamnesis? Comparative evaluation of ChatGPT, Claude, and Gemini in diagnostic reasoning through case-based questioning in oral and maxillofacial disorders. J Stomatol Oral Maxillofac Surg. 2026;127:102644. https:\/\/doi.org\/10.1016\/j.jormas.2025.102644.","journal-title":"J Stomatol Oral Maxillofac Surg"},{"key":"9182_CR22","doi-asserted-by":"publisher","first-page":"204209862513393","DOI":"10.1177\/20420986251339358","volume":"16","author":"J Sicard","year":"2025","unstructured":"Sicard J, Montastruc F, Achalme C, Jonville-Bera AP, Songue P, Babin M, et al. Can large language models detect drug-drug interactions leading to adverse drug reactions? Ther Adv Drug Saf. 2025;16:20420986251339358. https:\/\/doi.org\/10.1177\/20420986251339358.","journal-title":"Ther Adv Drug Saf"},{"key":"9182_CR23","doi-asserted-by":"publisher","first-page":"1495582","DOI":"10.3389\/fmed.2024.1495582","volume":"11","author":"J-C B\u00e9lisle-Pipon","year":"2024","unstructured":"B\u00e9lisle-Pipon J-C. Why we need to be careful with LLMs in medicine. Front Med (Lausanne). 2024;11:1495582. https:\/\/doi.org\/10.3389\/fmed.2024.1495582.","journal-title":"Front Med (Lausanne)"},{"key":"9182_CR24","doi-asserted-by":"publisher","first-page":"e90","DOI":"10.1017\/ash.2025.47","volume":"5","author":"S AlGain","year":"2025","unstructured":"AlGain S, Marra AR, Kobayashi T, Marra PS, Celeghini PD, Hsieh MK, et al. Can we rely on artificial intelligence to guide antimicrobial therapy? A systematic literature review. Antimicrob Steward Healthc Epidemiol. 2025;5:e90. https:\/\/doi.org\/10.1017\/ash.2025.47.","journal-title":"Antimicrob Steward Healthc Epidemiol"},{"key":"9182_CR25","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1038\/s44259-025-00084-5","volume":"3","author":"DR Giacobbe","year":"2025","unstructured":"Giacobbe DR, Marelli C, La Manna B, Padua D, Malva A, Guastavino S, et al. Advantages and limitations of large language models for antibiotic prescribing and antimicrobial stewardship. npj Antimicrob Resist. 2025;3:14. https:\/\/doi.org\/10.1038\/s44259-025-00084-5.","journal-title":"npj Antimicrob Resist"},{"key":"9182_CR26","doi-asserted-by":"publisher","first-page":"61","DOI":"10.2340\/aos.v85.45419","volume":"85","author":"F Ozbey","year":"2026","unstructured":"Ozbey F, Gokkurt Yilmaz BN, Yilmaz BE. Dentists\u2019 awareness of data security and ethical issues during the transition to artificial intelligence-driven clinical practice. Acta Odontol Scand. 2026;85:61\u20136. https:\/\/doi.org\/10.2340\/aos.v85.45419.","journal-title":"Acta Odontol Scand"}],"container-title":["BMC Oral Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12903-026-09182-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12903-026-09182-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12903-026-09182-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T14:03:15Z","timestamp":1790258595000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s12903-026-09182-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,4]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["9182"],"URL":"https:\/\/doi.org\/10.1186\/s12903-026-09182-w","relation":{},"ISSN":["1472-6831"],"issn-type":[{"value":"1472-6831","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,4]]},"assertion":[{"value":"5 May 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 June 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 July 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This study did not involve human participants, patient data, animal subjects, or any clinical intervention. All data were generated through interactions with publicly accessible large language model platforms (ChatGPT 5.3, Gemini 3.1 Pro, and Claude 4.6 Sonnet) via their official web interfaces. Formal ethics committee approval was therefore not required. The study was conducted in accordance with the principles outlined in the Declaration of Helsinki.","order":1,"name":"Ethics","label":"Ethics approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"1808"}}