{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T00:30:07Z","timestamp":1790814607728,"version":"4.1.0"},"reference-count":41,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,6,21]],"date-time":"2025-06-21T00:00:00Z","timestamp":1750464000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Dentistry Journal"],"abstract":"<jats:p>Background\/Objectives: This study aims to evaluate the response accuracy and quality of three AI chatbots\u2014GPT-4.0, Claude-2, and Llama-2\u2014in answering multiple-choice questions in prosthodontic and restorative dentistry. Methods: A total of 191 text-based multiple-choice questions were selected from the prosthodontic and restorative dentistry sections of the United States Integrated National Board Dental Examination (INBDE) (n = 80) and the United Kingdom Overseas Registration Examination (ORE) (n = 111). These questions were inputted into the chatbots, and the AI-generated answers were compared with the official answer keys to determine their accuracy. Additionally, two dental specialists independently evaluated the rationales accompanying each chatbot response for accuracy, relevance, and comprehensiveness, categorizing them into four distinct ratings. Chi-square and post hoc Z-tests with Bonferroni adjustment were used to analyze the responses. The inter-rater reliability for evaluating the quality of the rationale ratings among specialists was assessed using Cohen\u2019s Kappa (\u03ba). Results: GPT-4.0 (65.4%; n = 125\/191) demonstrated a significantly higher proportion of correctly answered multiple-choice questions when compared to Claude-2 (41.9%; n = 80\/191) (p &lt; 0.017) and Llama-2 (26.2%; n = 50\/191) (p &lt; 0.017). Significant differences were observed in the answer accuracy among all of the chatbots (p &lt; 0.001). In terms of the rationale quality, GPT-4.0 (58.1%; n = 111\/191) had a significantly higher proportion of \u201cCorrect Answer, Correct Rationale\u201d than Claude-2 (37.2%; n = 71\/191) (p &lt; 0.017) and Llama-2 (24.1%; n = 46\/191) (p &lt; 0.017). Significant differences were observed in the rationale quality among all of the chatbots (p &lt; 0.001). The inter-rater reliability was very high (\u03ba = 0.83). Conclusions: GPT-4.0 demonstrated the highest accuracy and quality of reasoning in responding to prosthodontic and restorative dentistry questions. This underscores the varying efficacy of AI chatbots within specialized dental contexts.<\/jats:p>","DOI":"10.3390\/dj13070279","type":"journal-article","created":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T04:50:05Z","timestamp":1750654205000},"page":"279","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":40,"title":["Evaluation of Chatbot Responses to Text-Based Multiple-Choice Questions in Prosthodontic and Restorative Dentistry"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5691-6806","authenticated-orcid":false,"given":"Reinhard Chun Wang","family":"Chau","sequence":"first","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8710-1709","authenticated-orcid":false,"given":"Khaing Myat","family":"Thu","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5310-5039","authenticated-orcid":false,"given":"Ollie Yiru","family":"Yu","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7699-1937","authenticated-orcid":false,"given":"Richard Tai-Chiu","family":"Hsung","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"},{"name":"Department of Computer Science, Hong Kong Chu Hai College, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4208-8373","authenticated-orcid":false,"given":"Denny Chon Pei","family":"Wang","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8665-6108","authenticated-orcid":false,"given":"Manuel Wing Ho","family":"Man","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John Junwen","family":"Wang","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5530-2645","authenticated-orcid":false,"given":"Walter Yu Hang","family":"Lam","sequence":"additional","affiliation":[{"name":"Faculty of Dentistry, The University of Hong Kong, Hong Kong 999077, China"},{"name":"Musketeers Foundation Institute of Data Science, The University of Hong Kong, Hong Kong 999077, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"84486","DOI":"10.1109\/ACCESS.2022.3197671","article-title":"Explainable AI for healthcare 5.0: Opportunities and challenges","volume":"10","author":"Saraswat","year":"2022","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"100025","DOI":"10.1016\/j.caeai.2021.100025","article-title":"AI technologies for education: Recent research & future directions","volume":"2","author":"Zhang","year":"2021","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1007\/s10439-023-03172-7","article-title":"Role of Chat GPT in public health","volume":"51","author":"Biswas","year":"2023","journal-title":"Ann. Biomed. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e179","DOI":"10.1016\/S2589-7500(23)00048-1","article-title":"Using ChatGPT to write patient clinic letters","volume":"5","author":"Ali","year":"2023","journal-title":"Lancet Digit. Health"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"792","DOI":"10.1001\/jama.2023.14311","article-title":"Large Language Models Answer Medical Questions Accurately, but Can\u2019t Match Clinicians\u2019 Knowledge","volume":"330","author":"Harris","year":"2023","journal-title":"JAMA"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1082","DOI":"10.1016\/j.prosdent.2023.05.023","article-title":"How much can we rely on artificial intelligence chatbots such as the ChatGPT software program to assist with scientific writing?","volume":"133","author":"Dashti","year":"2023","journal-title":"J. Prosthet. Dent."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"The Lancet Regional Health\u2013Europe (2023). Embracing Generative AI in Health Care, Elsevier.","DOI":"10.1016\/j.lanepe.2023.100677"},{"key":"ref_8","unstructured":"(2023, April 20). INBDE\u00ae History and Purpose. Available online: https:\/\/jcnde.ada.org\/inbde\/inbde-history."},{"key":"ref_9","unstructured":"General Dental Council (2022). Overseas Registration Examination Part 1 Results for August 2022."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1038\/sj.bdj.2009.1011","article-title":"Designing a dental curriculum for the twenty-first century","volume":"207","author":"McHarg","year":"2009","journal-title":"Br. Dent. J."},{"key":"ref_11","unstructured":"Klineberg, I., and Eckert, S. (2015). Functional Occlusion in Restorative Dentistry and Prosthodontics, Elsevier."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1016\/j.jds.2020.05.022","article-title":"Scope and performance of artificial intelligence technology in orthodontic diagnosis, treatment planning, and clinical decision-making-a systematic review","volume":"16","author":"Khanagar","year":"2021","journal-title":"J. Dent. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Thurzo, A., Strunga, M., Urban, R., Surovkov\u00e1, J., and Afrashtehfar, K.I. (2023). Impact of artificial intelligence on dental education: A review and guide for curriculum update. Educ. Sci., 13.","DOI":"10.3390\/educsci13020150"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1016\/j.identj.2023.12.007","article-title":"Performance of Generative Artificial Intelligence in Dental Licensing Examinations","volume":"74","author":"Chau","year":"2024","journal-title":"Int. Dent. J."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"575","DOI":"10.1007\/s00784-024-05968-w","article-title":"Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments","volume":"28","author":"Paris","year":"2024","journal-title":"Clin. Oral Investig."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1111\/iej.13985","article-title":"Unveiling the ChatGPT phenomenon: Evaluating the consistency and accuracy of endodontic question answers","volume":"57","author":"Algar","year":"2024","journal-title":"Int. Endod. J."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"104938","DOI":"10.1016\/j.jdent.2024.104938","article-title":"Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: A pilot study","volume":"144","author":"Rokhshad","year":"2024","journal-title":"J. Dent."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"158","DOI":"10.3390\/ai5010009","article-title":"AI and Face-Driven Orthodontics: A Scoping Review of Digital Advances in Diagnosis and Treatment Planning","volume":"5","author":"Zsoldos","year":"2024","journal-title":"AI"},{"key":"ref_19","unstructured":"(2023, April 20). QS World University Rankings by Subject 2023: Dentistry. Available online: https:\/\/www.topuniversities.com\/university-rankings\/university-subject-rankings\/2023\/dentistry."},{"key":"ref_20","unstructured":"Dowd, F.J. (2007). Mosby\u2019s Review for the NBDE Part Two, Mosby Elsevier."},{"key":"ref_21","unstructured":"(2023, April 20). Amazon.com: US NBDE Book. Available online: https:\/\/www.amazon.com\/s?k=US+nbde+books&crid=254FFUWU4IBTB&sprefix=usnbde+books%2Caps%2C456&ref=nb_sb_noss."},{"key":"ref_22","unstructured":"Hammond, D. (2014). Best of Fives for Dentistry, PasTest Ltd.. [3rd ed.]."},{"key":"ref_23","unstructured":"Fan, K.F.M., Jones, J., and Quinn, B.F.A. (2014). MCQs for Dentistry, PasTest Ltd.. [3rd ed.]."},{"key":"ref_24","unstructured":"(2023, April 20). Books by Pastest. Available online: https:\/\/www.bookdepository.com\/publishers\/Pastest."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1080\/08824096.2024.2327547","article-title":"From manual to machine: Assessing the efficacy of large language models in content analysis","volume":"41","author":"Pilny","year":"2024","journal-title":"Commun. Res. Rep."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"104796","DOI":"10.1016\/j.jdent.2023.104796","article-title":"Natural Language Processing: Chances and Challenges in Dentistry","volume":"141","author":"Leser","year":"2024","journal-title":"J. Dent."},{"key":"ref_27","unstructured":"Quora (2025, May 08). About Poe. Available online: https:\/\/poe.com\/about."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"S102","DOI":"10.1016\/j.jds.2024.10.012","article-title":"The virtual assessment in dental education: A narrative review","volume":"19","author":"Lin","year":"2024","journal-title":"J. Dent. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e51580","DOI":"10.2196\/51580","article-title":"Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: Comparative mixed methods study","volume":"25","author":"Giannakopoulos","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_30","first-page":"360","article-title":"Understanding interobserver agreement: The kappa statistic","volume":"37","author":"Viera","year":"2005","journal-title":"Fam. Med."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"19","DOI":"10.2186\/jpr.JPR_D_20_00139","article-title":"Application of artificial intelligence in the dental field: A literature review","volume":"66","author":"Kishimoto","year":"2022","journal-title":"J. Prosthodont. Res."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1970","DOI":"10.1016\/j.identj.2025.01.008","article-title":"External Validation of an AI mHealth Tool for Gingivitis Detection among Older Adults at Daycare Centers: A Pilot Study","volume":"75","author":"Chau","year":"2025","journal-title":"Int. Dent. J."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1177\/0022034520969115","article-title":"Application of artificial intelligence in dentistry","volume":"100","author":"Shan","year":"2021","journal-title":"J. Dent. Res."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Gheisarifar, M., Shembesh, M., Koseoglu, M., Fang, Q., Afshari, F.S., Yuan, J.C.-C., and Sukotjo, C. (2025). Evaluating the validity and consistency of artificial intelligence chatbots in responding to patients\u2019 frequently asked questions in prosthodontics. J. Prosthet. Dent.","DOI":"10.1016\/j.prosdent.2025.03.009"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1740","DOI":"10.1111\/jerd.13447","article-title":"Evaluating the Accuracy, Reliability, Consistency, and Readability of Different Large Language Models in Restorative Dentistry","volume":"37","author":"Ozdemir","year":"2025","journal-title":"J. Esthet. Restor. Dent."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"100074","DOI":"10.1016\/j.caeai.2022.100074","article-title":"Explainable artificial intelligence in education","volume":"3","author":"Khosravi","year":"2022","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1002\/JPER.23-0514","article-title":"Artificial intelligence in dental education: ChatGPT\u2019s performance on the periodontic in-service examination","volume":"95","author":"Danesh","year":"2024","journal-title":"J. Periodontol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"e54985","DOI":"10.2196\/54985","article-title":"The Diagnostic Ability of GPT-3.5 and GPT-4.0 in Surgery: Comparative Analysis","volume":"26","author":"Liu","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kipp, M. (2024). From GPT-3.5 to GPT-4.o: A Leap in AI\u2019s Medical Exam Performance. Information, 15.","DOI":"10.3390\/info15090543"},{"key":"ref_40","first-page":"327","article-title":"Self-monitoring of Oral Health Using Smartphone Selfie Powered by Artificial Intelligence: Implications for Preventive Dentistry","volume":"22","author":"Chau","year":"2024","journal-title":"Oral Health Prev. Dent."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"769","DOI":"10.1177\/0022034520915714","article-title":"Artificial intelligence in dentistry: Chances and challenges","volume":"99","author":"Schwendicke","year":"2020","journal-title":"J. Dent. Res."}],"container-title":["Dentistry Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2304-6767\/13\/7\/279\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:56:24Z","timestamp":1760032584000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2304-6767\/13\/7\/279"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,21]]},"references-count":41,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["dj13070279"],"URL":"https:\/\/doi.org\/10.3390\/dj13070279","relation":{},"ISSN":["2304-6767"],"issn-type":[{"value":"2304-6767","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,21]]}}}