{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T13:59:20Z","timestamp":1793627960284,"version":"4.5.1"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T00:00:00Z","timestamp":1770336000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T00:00:00Z","timestamp":1770854400000},"content-version":"vor","delay-in-days":6,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Galala University"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Oral Health"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The automated grading of clinical short-answer questions using large language models (LLMs) could alleviate faculty workload and improve the immediacy of feedback in dental education. However, evidence on the capacity of LLMs for rubric-based grading in dentistry remains limited. Therefore, this study aimed to compare the grading reliability and error patterns of two LLMs, ChatGPT-4 and the open-weight DeepSeek-3, against expert human evaluators.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>In a retrospective cross-sectional study with comparative validation design, we analyzed 2,358 short-answer responses from 262 undergraduate dental students (across nine clinical questions). All responses were analyzed, then human-graded by three calibrated subject-matter experts (SME) (intraclass correlation coefficient [ICC]\u2009=\u20090.84) to provide a reference. Each LLM was provided a 12-point analytic rubric to guide the grading, but was not provided any prior examples of the grading task (i.e., a zero-shot prompt). We assessed agreement using ICC, Pearson correlation, Cohen\u2019s kappa, and mixed-effects models, and examined error tiers (\u2264\u20091, 2\u20133, &gt;\u20093 points) across Bloom\u2019s levels and response styles.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      In this dataset, DeepSeek-3 obtained an ICC of 0.87 compared with ChatGPT-4 which obtained an ICC of 0.64. DeepSeek-3 matched exactly with human scores in 43.3% of cases and was within \u00b1\u20091 point in 62.4%, compared with 35.5% and 44.1%\u2002for ChatGPT-4. High-error\u2002rates (&gt;\u20093 points) were 7.5% for DeepSeek-3 vs. 26.9% for ChatGPT-4 (\u03c7\u00b2,\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.01). DeepSeek-3\u2019s agreement was consistent across cognitive levels and response verbosity, while ChatGPT-4\u2019s accuracy on higher-level and verbose responses was significantly lower (\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.01). Both models exhibited an optimistic bias by over-scoring incorrect answers.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>DeepSeek-3 showed fewer large-magnitude errors and better agreement with human graders compared to ChatGPT-4, suggesting its potential value for large-scale AI-assisted assessment for dental education. Since both models can over-score on incorrect results, human-in-the-loop oversight is necessary for\u2002high-stakes applications. Further work should evaluate performance across more courses, institutions, and languages, as well as examine the effects of model calibration, bias reduction, and external validation before considering the broader integration of LLMs\u2002into dental education.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12903-026-07665-4","type":"journal-article","created":{"date-parts":[[2026,2,6]],"date-time":"2026-02-06T10:15:44Z","timestamp":1770372944000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Calibration of AI large language models with human subject matter experts for grading of clinical short-answer responses in dental education"],"prefix":"10.1186","volume":"26","author":[{"given":"Fatma E.A.","family":"Hassanein","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Radwa R.","family":"Hussein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yousra","family":"Ahmed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jylan","family":"El-Guindy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Doha E.","family":"Ahmed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5069-8257","authenticated-orcid":false,"given":"Asmaa","family":"Abou-Bakr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,2,6]]},"reference":[{"key":"7665_CR1","doi-asserted-by":"publisher","unstructured":"Vrdoljak J, Boban Z, Vilovi\u0107 M, Kumri\u0107 M, Bo\u017ei\u0107 J. A review of large language models in medical education, clinical decision support, and healthcare administration. Healthc (Basel). 2025;13(6):603. https:\/\/doi.org\/10.3390\/healthcare13060603.","DOI":"10.3390\/healthcare13060603"},{"key":"7665_CR2","doi-asserted-by":"crossref","unstructured":"Morjaria L, Burns L, Bracken K, Levinson AJ, Ngo QN, Lee M, Sibbald M. Examining the Efficacy of ChatGPT in Marking Short-Answer Assessments in an Undergraduate Medical Program.\u00a0Int Med Educ.\u00a02024;3:32\u201343.","DOI":"10.3390\/ime3010004"},{"issue":"1","key":"7665_CR3","doi-asserted-by":"publisher","first-page":"856","DOI":"10.1016\/j.identj.2025.109357","volume":"76","author":"A Robaian","year":"2026","unstructured":"Robaian A, Hassanein EA, Hassan MT, Alqahtani AS, Abou-Bakr A. A multimodal large language model framework for clinical subtyping and malignant transformation risk prediction in oral lichen planus: a paired comparison with expert clinicians. Int Dent J. 2026;76.","journal-title":"International Dental Journal"},{"key":"7665_CR4","doi-asserted-by":"publisher","first-page":"e52346","DOI":"10.2196\/52346","volume":"10","author":"D Claman","year":"2024","unstructured":"Claman D, Sezgin E. Artificial intelligence in dental education: opportunities and challenges of large language models and multimodal foundation models. JMIR Med Educ. 2024;10:e52346.","journal-title":"JMIR Med Educ"},{"issue":"1","key":"7665_CR5","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1002\/jdd.13688","volume":"89","author":"R Delgado-Ruiz","year":"2025","unstructured":"Delgado-Ruiz R, Kim AS, Zhang H, Sullivan D, Awan KH, Stathopoulou PG. Generative artificial intelligence (Gen AI) in dental education: opportunities, cautions, and recommendations. J Dent Educ. 2025;89(1):130\u20136.","journal-title":"J Dent Educ"},{"issue":"1","key":"7665_CR6","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.1186\/s12909-024-06026-5","volume":"24","author":"C Gr\u00e9visse","year":"2024","unstructured":"Gr\u00e9visse C. LLM-based automatic short answer grading in undergraduate medical education. BMC Med Educ. 2024;24(1):1060.","journal-title":"BMC Med Educ"},{"key":"7665_CR7","doi-asserted-by":"publisher","unstructured":"Hassanein FEA, El Barbary A, Hussein RR, Ahmed Y, El-Guindy J, Sarhan S, et al. Diagnostic performance of ChatGPT-4o and DeepSeek-3 differential diagnosis of complex oral lesions: a multimodal imaging and case difficulty analysis. Oral Dis. 2025. Epub ahead of print. https:\/\/doi.org\/10.1111\/odi.70007.","DOI":"10.1111\/odi.70007"},{"issue":"1","key":"7665_CR8","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1186\/s12909-020-02057-w","volume":"20","author":"T Puthiaparampil","year":"2020","unstructured":"Puthiaparampil T, Rahman MM. Very short answer questions: a viable alternative to multiple choice questions. BMC Med Educ. 2020;20(1):141.","journal-title":"BMC Med Educ"},{"key":"7665_CR9","doi-asserted-by":"publisher","first-page":"105091","DOI":"10.1016\/j.jdent.2024.105091","volume":"148","author":"C Louca","year":"2024","unstructured":"Louca C, Fine P, Tonni I, Leung A. Reviewing assessment strategies in European dental schools. J Dent. 2024;148:105091.","journal-title":"J Dent"},{"issue":"1","key":"7665_CR10","doi-asserted-by":"publisher","first-page":"37932","DOI":"10.1038\/s41598-025-22979-z","volume":"15","author":"FEA Hassanein","year":"2025","unstructured":"Hassanein FEA, Ahmed Y, Maher S, Barbary AE, Abou-Bakr A. Prompt-dependent performance of multimodal AI model in oral diagnosis: a comprehensive analysis of accuracy, narrative quality, calibration, and latency versus human experts. Sci Rep. 2025;15(1):37932.","journal-title":"Sci Rep"},{"issue":"1","key":"7665_CR11","doi-asserted-by":"publisher","first-page":"43279","DOI":"10.1038\/s41598-025-28862-1","volume":"15","author":"FEA Hassanein","year":"2025","unstructured":"Hassanein FEA, Hussein RR, Almalahy HG, Sarhan S, Ahmed Y, Abou-Bakr A. Vision-based diagnostic gain of ChatGPT-5 and gemini 2.5 pro compared with human experts in oral lesion assessment. Sci Rep. 2025;15(1):43279.","journal-title":"Sci Rep"},{"key":"7665_CR12","doi-asserted-by":"publisher","unstructured":"Abou-Bakr A, El Barbary A, Hassanein FEA. ChatGPT-5 vs oral medicine experts for rank-based differential diagnosis of oral lesions: a prospective, biopsy-validated comparison. Odontology. 2025. Epub ahead of print. https:\/\/doi.org\/10.1007\/s10266-025-01242-x.","DOI":"10.1007\/s10266-025-01242-x"},{"key":"7665_CR13","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I. Attention is all you need. Adv Neural Inf Process Syst. 2017;30."},{"key":"7665_CR14","doi-asserted-by":"publisher","unstructured":"Shermis MD. Using ChatGPT to score essays and Short-Form constructed responses. Assessing Writing. 2025;66. https:\/\/doi.org\/10.1016\/j.asw.2025.100988.","DOI":"10.1016\/j.asw.2025.100988"},{"issue":"1","key":"7665_CR15","doi-asserted-by":"publisher","first-page":"e0317423","DOI":"10.1371\/journal.pone.0317423","volume":"20","author":"HC Nguyen","year":"2025","unstructured":"Nguyen HC, Dang HP, Nguyen TL, Hoang V, Nguyen VA. Accuracy of latest large language models in answering multiple choice questions in dentistry: a comparative study. PLoS One. 2025;20(1):e0317423.","journal-title":"PLoS One"},{"key":"7665_CR16","doi-asserted-by":"publisher","unstructured":"Xiao D, Gao C, Luo Z, Liu C, Shen S. Can LLMs assist computer education? An empirical case study of deepseek. In: Zhu T, Zhou W, Zhu C, editors. Knowledge Science, Engineering and Management. KSEM 2025. Lecture Notes in Computer Science(), vol 15920. Singapore; Springer. https:\/\/doi.org\/10.1007\/978-981-95-3052-6_29.","DOI":"10.1007\/978-981-95-3052-6_29"},{"issue":"5","key":"7665_CR17","first-page":"337","volume":"46","author":"R Rokhshad","year":"2024","unstructured":"Rokhshad R, Fadul M, Zhai G, Carr K, Jackson JG, Zhang P. A comparative analysis of responses of artificial intelligence chatbots in special needs dentistry. Pediatr Dent. 2024;46(5):337\u201344.","journal-title":"Pediatr Dent"},{"issue":"3","key":"7665_CR18","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1016\/j.identj.2023.12.007","volume":"74","author":"RCW Chau","year":"2024","unstructured":"Chau RCW, Thu KM, Yu OY, Hsung RT-C, Lo ECM, Lam WYH. Performance of generative artificial intelligence in dental licensing examinations. Int Dent J. 2024;74(3):616\u201321.","journal-title":"Int Dent J"},{"key":"7665_CR19","doi-asserted-by":"publisher","first-page":"104938","DOI":"10.1016\/j.jdent.2024.104938","volume":"144","author":"R Rokhshad","year":"2024","unstructured":"Rokhshad R, Zhang P, Mohammad-Rahimi H, Pitchika V, Entezari N, Schwendicke F. Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: A pilot study. J Dent. 2024;144:104938.","journal-title":"J Dent"},{"key":"7665_CR20","doi-asserted-by":"publisher","unstructured":"Chau RC, Thu KM, Yu OY, Hsung RT, Wang DC, Man MW, et al. Evaluation of chatbot responses to text-based multiple-choice questions in prosthodontic and restorative dentistry.\u00a0Dent J. 2025;13:279. https:\/\/doi.org\/10.3390\/dj13070279.","DOI":"10.3390\/dj13070279"},{"key":"7665_CR21","doi-asserted-by":"crossref","unstructured":"Abou-Bakr A, Eissa AA, Alshikh B, Ahmed Y, AbuShady EF, Tassoker M, Hassanein FEA. Comparative diagnostic accuracy of ChatGPT models in salivary gland disease: a multimodal vignette-based evaluation.\u00a0Eur Arch Otorhinolaryngol. 2025.","DOI":"10.1007\/s00405-025-09925-5"},{"key":"7665_CR22","doi-asserted-by":"publisher","first-page":"e45312","DOI":"10.2196\/45312","volume":"9","author":"A Gilson","year":"2023","unstructured":"Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the united States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9:e45312.","journal-title":"JMIR Med Educ"},{"key":"7665_CR23","doi-asserted-by":"publisher","first-page":"e56342","DOI":"10.2196\/56342","volume":"10","author":"HB Burke","year":"2024","unstructured":"Burke HB, Hoang A, Lopreiato JO, King H, Hemmer P, Montgomery M, et al. Assessing the ability of a large language model to score free-text medical student clinical notes: quantitative study. JMIR Med Educ. 2024;10:e56342.","journal-title":"JMIR Med Educ"},{"issue":"1","key":"7665_CR24","doi-asserted-by":"publisher","first-page":"32","DOI":"10.3390\/ime3010004","volume":"3","author":"L Morjaria","year":"2024","unstructured":"Morjaria L, Burns L, Bracken K, Levinson AJ, Ngo QN, Lee M, et al. Examining the efficacy of ChatGPT in marking short-answer assessments in an undergraduate medical program. Int J Med Educ. 2024;3(1):32\u201343.","journal-title":"Int J Med Educ"},{"key":"7665_CR25","unstructured":"Anderson L, Krathwohl D, Bloom B. A Taxonomy for Learning, Teaching, and Assessing: a Revision of Bloom\u2019s Taxonomy of Educational Objectives. 2000. http:\/\/lst-iiepiiep-unescoorg\/cgi-bin\/wwwi32exe\/[in=epidoc1in]\/?t2000=015556\/(100).\u00a0"},{"key":"7665_CR26","doi-asserted-by":"crossref","unstructured":"Ammar K, Rais RNB. Outcome Based Performance Evaluation Approach Using Rubric Based Capstone Projects Assessment. In: 2023 20th International Multi-Conference on Systems, Signals & Devices (SSD). 2023;2023:777\u2013783.","DOI":"10.1109\/SSD58187.2023.10411275"},{"key":"7665_CR27","doi-asserted-by":"crossref","unstructured":"Bisante A, Dix A, Panizzi E, Zeppieri S. To err is AI. In: Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter. Torino, Italy: Association for Computing Machinery; 2023: Article 13.","DOI":"10.1145\/3605390.3605414"},{"key":"7665_CR28","doi-asserted-by":"publisher","unstructured":"Hackl V, M\u00fcller AE, Granitzer M, Sailer M. Is GPT-4 a reliable rater? Evaluating consistency in GPT-4 text ratings. Frontiers in Education. 2023;8. https:\/\/doi.org\/10.3389\/feduc.2023.1272229.","DOI":"10.3389\/feduc.2023.1272229"},{"key":"7665_CR29","doi-asserted-by":"publisher","unstructured":"Ade-Ibijola A, Chikezie IJ, Oyelere SS. Human vs. Machine marking: a comparative study of chemistry assessments. J Sci Educ Technol. 2025;34:1430\u201340. https:\/\/doi.org\/10.1007\/s10956-025-10223-2.","DOI":"10.1007\/s10956-025-10223-2"},{"issue":"1","key":"7665_CR30","doi-asserted-by":"publisher","first-page":"962","DOI":"10.1186\/s12909-024-05881-6","volume":"24","author":"B Quah","year":"2024","unstructured":"Quah B, Zheng L, Sng TJH, Yong CW, Islam I. Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations. BMC Med Educ. 2024;24(1):962.","journal-title":"BMC Med Educ"},{"key":"7665_CR31","unstructured":"Jade T, Yartsev A. ChatGPT for automated grading of short answer questions in mechanical ventilation. ArXiv Preprint arXiv. 2025:250504645."},{"key":"7665_CR32","doi-asserted-by":"publisher","unstructured":"Dincer HA, Dogu D. Evaluating artificial intelligence in patient education: DeepSeek-V3 versus ChatGPT-4o in answering common questions on laparoscopic cholecystectomy. ANZ J Surg. 2025;95(11):2322\u20138. https:\/\/doi.org\/10.1111\/ans.70198.","DOI":"10.1111\/ans.70198"},{"key":"7665_CR33","doi-asserted-by":"publisher","unstructured":"Armfield D, et al. Avalon: a human-in-the-loop LLM grading system with instructor calibration and student self-assessment. In: Cristea AI, Walker E, Lu Y, Santos OC, Isotani S, Editors. Artificial intelligence in education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium, Blue Sky, and WideAIED. AIED 2025. Communications in Computer and Information Science, vol 2592. Cham; Springer. https:\/\/doi.org\/10.1007\/978-3-031-99267-4_14.","DOI":"10.1007\/978-3-031-99267-4_14"},{"key":"7665_CR34","doi-asserted-by":"publisher","unstructured":"Poli\u010dar PG, \u0160pendl M, Curk T, Zupan B. Automated assignment grading with large Language models: insights from a bioinformatics course. Bioinformatics. 2025;41(1):i21\u20139. https:\/\/doi.org\/10.1093\/bioinformatics\/btaf196.","DOI":"10.1093\/bioinformatics\/btaf196"},{"key":"7665_CR35","doi-asserted-by":"publisher","first-page":"107523","DOI":"10.1016\/j.infsof.2024.107523","volume":"175","author":"C Pornprasit","year":"2024","unstructured":"Pornprasit C, Tantithamthavorn C. Fine-tuning and prompt engineering for large Language models-based code review automation. \u200eInf Softw Technol. 2024;175:107523.","journal-title":"\u200eInf Softw Technol"},{"key":"7665_CR36","doi-asserted-by":"crossref","unstructured":"Yeung C, Yu J, Cheung K, Wong T, Chan C, Wong K, Fujii K. A Zero-Shot LLM framework. for Automatic Assignment Grading in Higher Education; 2025.","DOI":"10.1007\/978-3-031-99264-3_3"},{"key":"7665_CR37","doi-asserted-by":"crossref","unstructured":"Blodgett S, Barocas S, Daum\u00e9 III, Wallach H. Language (Technology) is power. A Critical Survey of Bias in NLP; 2020.","DOI":"10.18653\/v1\/2020.acl-main.485"},{"issue":"6464","key":"7665_CR38","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","volume":"366","author":"Z Obermeyer","year":"2019","unstructured":"Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447\u201353.","journal-title":"Science"}],"container-title":["BMC Oral Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12903-026-07665-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12903-026-07665-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12903-026-07665-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T17:40:01Z","timestamp":1781631601000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s12903-026-07665-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,6]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["7665"],"URL":"https:\/\/doi.org\/10.1186\/s12903-026-07665-4","relation":{},"ISSN":["1472-6831"],"issn-type":[{"value":"1472-6831","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,6]]},"assertion":[{"value":"5 August 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 January 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethical approval was obtained from the Research Ethics Committee of the Faculty of Dentistry, Ain Shams University (Approval No. FDASU-RecER072508). As this study involved retrospective analysis of anonymized student assessment data without any direct intervention, informed consent was waived by the Ethics Committee. All procedures involving human participants were conducted in accordance with the ethical standards of the institutional research committee and with the principles of the Declaration of Helsinki.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"286"}}