{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T09:08:48Z","timestamp":1789549728768,"version":"build-2803163510"},"reference-count":40,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T00:00:00Z","timestamp":1789516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Dent. J.","Dentistry Journal"],"abstract":"<jats:p>Background\/Objectives: Large language model (LLM) chatbots are increasingly used for dental information and decision support, yet their accuracy and short-term reproducibility in endodontics remain insufficiently established. This study compared five chatbots using open-ended questions derived from established AAE and ESE endodontic guidelines. Methods: Twenty-six guideline-based questions were content-validated by five endodontists using Lawshe\u2019s Content Validity Ratio. ChatGPT-4o, Gemini 2.5 Pro, DeepSeek-V3-0324, ScholarGPT (academic version built on OpenAI\u2019s GPT-4 architecture), and MedGebra GPT-4 answered each question across three days and three sessions per day, yielding 1170 responses. Two blinded endodontists scored responses on a 5-point guideline-concordance scale. Brunner\u2013Langer LD-F2 analyses assessed model and temporal effects, while weighted kappa evaluated response consistency. Results: The overall model effect was significant (p &lt; 0.001). ChatGPT-4o, Gemini 2.5 Pro, DeepSeek-V3-0324, and ScholarGPT showed comparable performance, whereas MedGebra GPT-4 performed significantly lower after Bonferroni correction. The day effect was not significant (p = 0.054), while the session effect (p = 0.048) and model \u00d7 session interaction (p = 0.030) were significant. Weighted kappa values varied across models and assessment days, ranging from 0.689\u20130.730 for Gemini 2.5 Pro, 0.606\u20130.662 for ChatGPT-4o, 0.520\u20130.645 for DeepSeek-V3-0324, 0.458\u20130.592 for ScholarGPT, and 0.240\u20130.739 for MedGebra GPT-4. Conclusions: Guideline-aligned performance and short-term reproducibility differed across the evaluated chatbots, showing that accuracy and consistency represent distinct aspects of performance. Repeated assessment captured variation missed by single-session testing. These findings support guideline-based evaluation and clinician verification when generative AI chatbots are used to provide endodontic information relevant to decision support or dental education.<\/jats:p>","DOI":"10.3390\/dj14090598","type":"journal-article","created":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T07:58:59Z","timestamp":1789545539000},"page":"598","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Accuracy and Short-Term Consistency of Generative AI Chatbots in Guideline-Based Endodontic Decision Support: A Multi-Model Benchmarking Study"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7546-2048","authenticated-orcid":false,"given":"Ilg\u0131n","family":"\u0130lgenli","sequence":"first","affiliation":[{"name":"Department of Endodontics, Faculty of Dentistry, Ege University, 35040 Izmir, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"}]},{"given":"Ezgi","family":"Avc\u0131","sequence":"additional","affiliation":[{"name":"Department of Endodontics, Faculty of Dentistry, Ege University, 35040 Izmir, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"validation","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"}]},{"given":"Timur","family":"K\u00f6se","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Medical Informatics, Faculty of Medicine, Ege University, 35040 Izmir, Turkey"},{"name":"Department of Basic Medical Sciences, Faculty of Medicine, Ege University, 35040 Izmir, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1007\/s11886-013-0441-8","article-title":"Artificial intelligence in medicine and cardiac imaging: Harnessing big data and advanced computing to provide personalized medical diagnosis and treatment","volume":"16","author":"Dilsizian","year":"2014","journal-title":"Curr. Cardiol. Rep."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2353","DOI":"10.1001\/jama.2016.17438","article-title":"Adapting to artificial intelligence","volume":"316","author":"Jha","year":"2016","journal-title":"JAMA"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"365","DOI":"10.4103\/ijd.IJD_421_20","article-title":"Artificial intelligence: How is it changing medical sciences and its future?","volume":"65","author":"Basu","year":"2020","journal-title":"Indian J. Dermatol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lee, D.-W., Kim, S.-Y., Jeong, S.-N., and Lee, J.-H. (2021). Artificial intelligence in fractured dental implant detection and classification: Evaluation using dataset from two dental hospitals. Diagnostics, 11.","DOI":"10.3390\/diagnostics11020233"},{"key":"ref_5","first-page":"1096","article-title":"Precise Identification of Oral Cancer Lesions Using Artificial Intelligence","volume":"316","author":"Wuttisarnwattana","year":"2024","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1080\/03014460.2022.2105396","article-title":"Dental age estimation of Malaysian Indian children and adolescents: Applicability of Chaillet and Demirjian\u2019s modified method using artificial neural network","volume":"49","author":"Bunyarit","year":"2022","journal-title":"Ann. Hum. Biol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1352","DOI":"10.1016\/j.joen.2021.06.003","article-title":"Artificial intelligence in endodontics: Current applications and future directions","volume":"47","author":"Aminoshariae","year":"2021","journal-title":"J. Endod."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Endres, M.G., Hillen, F., Salloumis, M., Sedaghat, A.R., Niehues, S.M., Quatela, O., Hanken, H., Smeets, R., Beck-Broichsitter, B., and Rendenbach, C. (2020). Development of a Deep Learning Algorithm for Periapical Disease Detection in Dental Radiographs. Diagnostics, 10.","DOI":"10.3390\/diagnostics10060430"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1016\/j.oooo.2021.01.018","article-title":"Artificial intelligence for detection of periapical lesions on intraoral radiographs: Comparison between convolutional neural networks and human observers","volume":"131","author":"Pauwels","year":"2021","journal-title":"Oral Surg. Oral Med. Oral Pathol. Oral Radiol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1016\/j.joen.2019.03.016","article-title":"Deep learning for the radiographic detection of apical lesions","volume":"45","author":"Ekert","year":"2019","journal-title":"J. Endod."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1111\/iej.13985","article-title":"Unveiling the ChatGPT phenomenon: Evaluating the consistency and accuracy of endodontic question answers","volume":"57","author":"Algar","year":"2024","journal-title":"Int. Endod. J."},{"key":"ref_12","first-page":"e54193","article-title":"The Role of Chatbot GPT Technology in Undergraduate Dental Education","volume":"16","author":"Thorat","year":"2024","journal-title":"Cureus"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1515\/dx-2024-0034","article-title":"Comparative analysis of diagnostic accuracy in endodontic assessments: Dental students vs. artificial intelligence","volume":"11","author":"Qutieshat","year":"2024","journal-title":"Diagnosis"},{"key":"ref_14","first-page":"e38317","article-title":"ChatGPT in Dentistry: A Comprehensive Review","volume":"15","author":"Alhaidry","year":"2023","journal-title":"Cureus"},{"key":"ref_15","first-page":"999","article-title":"Artificial intelligence chatbots in endodontic education-Concepts and potential applications","volume":"59","author":"Setzer","year":"2025","journal-title":"Int. Endod. J."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e48291","DOI":"10.2196\/48291","article-title":"Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions","volume":"9","author":"AlSaad","year":"2023","journal-title":"JMIR Med. Educ."},{"key":"ref_17","unstructured":"American Association of Endodontists (2019). Guide to Clinical Endodontics, American Association of Endodontists. [6th ed.]. Available online: https:\/\/www.aae.org\/specialty\/download\/guide-to-clinical-endodontics\/."},{"key":"ref_18","unstructured":"European Society of Endodontology (2025, May 05). ESE Publishes S3 Level Clinical Practice Guidelines. Available online: https:\/\/www.e-s-e.eu\/news-related-to-ese\/ese-news\/ese-publishes-s3-level-clinical-practice-guidelines\/."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1111\/iej.14163","article-title":"Unveiling the power of artificial intelligence for image-based diagnosis and treatment in endodontics: An ally or adversary?","volume":"58","author":"Fontenele","year":"2025","journal-title":"Int. Endod. J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1111\/jerd.13046","article-title":"Implications of large language models such as ChatGPT for dental medicine","volume":"35","author":"Eggmann","year":"2023","journal-title":"J. Esthet. Restor. Dent."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kung, T.H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepa\u00f1o, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., and Maningo, J. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit. Health, 2.","DOI":"10.1371\/journal.pdig.0000198"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1111\/aej.70040","article-title":"Clinical Relevance of Large Language Models in Endodontics: Diagnostic Appropriateness Based on 50 Simulated Case Scenarios","volume":"52","author":"Karaca","year":"2026","journal-title":"Aust. Endod. J."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.joen.2025.01.002","article-title":"Performance of 4 artificial intelligence chatbots in answering endodontic questions","volume":"51","author":"Abdulrab","year":"2025","journal-title":"J. Endod."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"\u00d6zbay, Y., Erdo\u011fan, D., and Din\u00e7er, G.A. (2025). Evaluation of the performance of large language models in clinical decision-making in endodontics. BMC Oral Health, 25.","DOI":"10.1186\/s12903-025-06050-x"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"B\u00fcy\u00fck\u00f6zer \u00d6zkan, H., Do\u011fan \u00c7ankaya, T., and K\u00f6l\u00fc\u015f, T. (2025). The Impact of Language Variability on Artificial Intelligence Performance in Regenerative Endodontics. Healthcare, 13.","DOI":"10.3390\/healthcare13101190"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1010","DOI":"10.1016\/j.joen.2026.01.009","article-title":"From Evidence-based Endodontics to Generative AI: A Comparative Study of 11 Large Language Models","volume":"52","author":"Moreno","year":"2026","journal-title":"J. Endod."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"The CHART Collaborative (2025). Reporting guideline for chatbot health advice studies: The CHART statement. JAMA Netw. Open, 8, e2530220.","DOI":"10.1001\/jamanetworkopen.2025.30220"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1111\/j.1744-6570.1975.tb01393.x","article-title":"A Quantitative Approach to Content Validity","volume":"28","author":"Lawshe","year":"1975","journal-title":"Pers. Psychol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1177\/0748175613513808","article-title":"Critical values for Lawshe\u2019s content validity ratio: Revisiting the original methods of calculation","volume":"47","author":"Ayre","year":"2014","journal-title":"Meas. Eval. Couns. Dev."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2070","DOI":"10.1111\/j.1572-0241.2007.01325.x","article-title":"A systematic review of patient inflammatory bowel disease information resources on the World Wide Web","volume":"102","author":"Bernard","year":"2007","journal-title":"Am. J. Gastroenterol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Erkal, D., Felek, T., Butean, O.P., and Er, K. (2025). Dens invaginatus as a diagnostic challenge: Evaluating large language models against expert endodontic reasoning. BMC Oral Health, 25.","DOI":"10.1186\/s12903-025-06987-z"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/j.athoracsur.2024.03.023","article-title":"Chatbot reliability in managing thoracic surgical clinical scenarios","volume":"118","author":"Platz","year":"2024","journal-title":"Ann. Thorac. Surg."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2328","DOI":"10.1016\/j.prosdent.2024.10.020","article-title":"Large language models in periodontology: Assessing their performance in clinically relevant questions","volume":"134","author":"Chatzopoulos","year":"2025","journal-title":"J. Prosthet. Dent."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"e51580","DOI":"10.2196\/51580","article-title":"Evaluation of the Performance of Generative AI Large Language Models ChatGPT, Google Bard, and Microsoft Bing Chat in Supporting Evidence-Based Dentistry: Comparative Mixed Methods Study","volume":"25","author":"Giannakopoulos","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.ajem.2024.07.043","article-title":"The accuracy of Gemini, GPT-4, and GPT-4o in ECG analysis: A comparison with cardiologists and emergency medicine specialists","volume":"84","year":"2024","journal-title":"Am. J. Emerg. Med."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1111\/edt.12999","article-title":"Evidence-based potential of generative artificial intelligence large language models on dental avulsion: ChatGPT versus Gemini","volume":"41","author":"Cankar","year":"2025","journal-title":"Dent. Traumatol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"101471","DOI":"10.1016\/j.jormas.2023.101471","article-title":"Can ChatGPT be used in oral and maxillofacial surgery?","volume":"124","author":"Balel","year":"2023","journal-title":"J. Stomatol. Oral Maxillofac. Surg."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"e70304","DOI":"10.1111\/jep.70304","article-title":"Evaluation of artificial intelligence conversational models in providing information on dental implants: A comparative analysis of ChatGPT, Gemini and MedGebra","volume":"31","author":"Kofos","year":"2025","journal-title":"J. Eval. Clin. Pract."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/s44445-025-00091-2","article-title":"Orthodontic knowledge assessment: A comparison of five AI Chatbots","volume":"38","author":"Hakami","year":"2026","journal-title":"Saudi Dent. J."}],"container-title":["Dentistry Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2304-6767\/14\/9\/598\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T08:49:23Z","timestamp":1789548563000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2304-6767\/14\/9\/598"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,16]]},"references-count":40,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["dj14090598"],"URL":"https:\/\/doi.org\/10.3390\/dj14090598","relation":{},"ISSN":["2304-6767"],"issn-type":[{"value":"2304-6767","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,16]]}}}