{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T14:28:26Z","timestamp":1792074506918,"version":"4.4.0"},"reference-count":48,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2024,12,17]],"date-time":"2024-12-17T00:00:00Z","timestamp":1734393600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background: Orofacial pain (OFP) encompasses a complex array of conditions affecting the face, mouth, and jaws, often leading to significant diagnostic challenges and high rates of misdiagnosis. Artificial intelligence, particularly large language models like GPT4 (OpenAI, San Francisco, CA, USA), offers potential as a diagnostic aid in healthcare settings. Objective: To evaluate the diagnostic accuracy of GPT4 in OFP cases as a clinical decision support system (CDSS) and compare its performance against treating clinicians, expert evaluators, medical students, and general practitioners. Methods: A total of 100 anonymized patient case descriptions involving diverse OFP conditions were collected. GPT4 was prompted to generate primary and differential diagnoses for each case using the International Classification of Orofacial Pain (ICOP) criteria. Diagnoses were compared to gold-standard diagnoses established by treating clinicians, and a scoring system was used to assess accuracy at three hierarchical ICOP levels. A subset of 24 cases was also evaluated by two clinical experts, two final-year medical students, and two general practitioners for comparative analysis. Diagnostic performance and interrater reliability were calculated. Results: GPT4 achieved the highest accuracy level (ICOP level 3) in 38% of cases, with an overall diagnostic performance score of 157 out of 300 points (52%). The model provided accurate differential diagnoses in 80% of cases (400 out of 500 points). In the subset of 24 cases, the model\u2019s performance was comparable to non-expert human evaluators but was surpassed by clinical experts, who correctly diagnosed 54% of cases at level 3. GPT4 demonstrated high accuracy in specific categories, correctly diagnosing 81% of trigeminal neuralgia cases at level 3. Interrater reliability between GPT4 and human evaluators was low (\u03ba = 0.219, p &lt; 0.001), indicating variability in diagnostic agreement. Conclusions: GPT4 shows promise as a CDSS for OFP by improving diagnostic accuracy and offering structured differential diagnoses. While not yet outperforming expert clinicians, GPT4 can augment diagnostic workflows, particularly in primary care or educational settings. Effective integration into clinical practice requires adherence to rigorous guidelines, thorough validation, and ongoing professional oversight to ensure patient safety and diagnostic reliability.<\/jats:p>","DOI":"10.3390\/diagnostics14242835","type":"journal-article","created":{"date-parts":[[2024,12,17]],"date-time":"2024-12-17T03:46:02Z","timestamp":1734407162000},"page":"2835","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Development and Evaluation of a GPT4-Based Orofacial Pain Clinical Decision Support System"],"prefix":"10.3390","volume":"14","author":[{"given":"Charlotte","family":"Vueghs","sequence":"first","affiliation":[{"name":"Department of Oral and Maxillofacial Surgery, University Hospitals Leuven, 3000 Leuven, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hamid","family":"Shakeri","sequence":"additional","affiliation":[{"name":"Department of Oral and Maxillofacial Surgery, University Hospitals Leuven, 3000 Leuven, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tara","family":"Renton","sequence":"additional","affiliation":[{"name":"Department of Oral Surgery, King\u2019s College London Dental Institute, London SE5 9RW, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frederic","family":"Van der Cruyssen","sequence":"additional","affiliation":[{"name":"Department of Oral and Maxillofacial Surgery, University Hospitals Leuven, 3000 Leuven, Belgium"},{"name":"OMFS-IMPATH Research Group, KU Leuven, 3000 Leuven, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1038\/nrd3226","article-title":"Pain market","volume":"9","author":"Melnikova","year":"2010","journal-title":"Nat. Rev. Drug Discov."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s12207-014-9206-y","article-title":"The Psychological Evaluation of Patients with Chronic Pain: A Review of BHI 2 Clinical and Forensic Interpretive Considerations","volume":"7","author":"Bruns","year":"2014","journal-title":"Psychol. Inj. Law"},{"key":"ref_3","unstructured":"Pietrasik, T. (2024). Migraine and Other Headache Disorders, World Health Organization."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1016\/S1474-4422(24)00038-3","article-title":"Global, regional, and national burden of disorders affecting the nervous system, 1990\u20132021: A systematic analysis for the Global Burden of Disease Study 2021","volume":"23","author":"Steinmetz","year":"2024","journal-title":"Lancet Neurol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2443","DOI":"10.1001\/jama.290.18.2443","article-title":"Lost Productive Time and Cost Due to Common Pain Conditions in the US Workforce","volume":"290","author":"Stewart","year":"2003","journal-title":"JAMA"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1136\/bmjqs-2013-002627","article-title":"The frequency of diagnostic errors in outpatient care: Estimations from three large observational studies involving US adult populations","volume":"23","author":"Singh","year":"2014","journal-title":"BMJ Qual. Saf."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"94","DOI":"10.5005\/jp-journals-10046-0051","article-title":"Why Chronic Pain Patients are Misdiagnosed 40 to 80% of the Time?","volume":"2","author":"Hendler","year":"2016","journal-title":"J. Recent Adv. Pain"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"672","DOI":"10.1136\/bmjqs-2012-001550","article-title":"25-Year summary of US malpractice claims for diagnostic errors 1986\u20132010: An analysis from the National Practitioner Data Bank","volume":"22","author":"Lee","year":"2013","journal-title":"BMJ Qual. Saf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2853","DOI":"10.2147\/JPR.S269329","article-title":"Challenges of Misdiagnosis and Suboptimal Treatment of Persistent Idiopathic Facial Pain and Atypical Odontalgia: A Retrospective Multi-Centric Cross-Sectional Investigation","volume":"13","author":"Xiao","year":"2020","journal-title":"J. Pain Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1543","DOI":"10.1111\/odi.14553","article-title":"Burning mouth syndrome: Analysis of diagnostic delay in 500 patients","volume":"30","author":"Adamo","year":"2024","journal-title":"Oral Dis."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"(2013). Headache Classification Committee of the International Headache Society (IHS) The International Classification of Headache Disorders, 3rd edition (beta version). Cephalalgia, 33, 629\u2013808.","DOI":"10.1177\/0333102413485658"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"(2020). International Classification of Orofacial Pain, 1st edition (ICOP). Cephalalgia, 40, 129\u2013221.","DOI":"10.1177\/0333102419893823"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Al-Antari, M.A. (2023). Artificial Intelligence for Medical Diagnostics\u2014Existing and Future AI Technology!. Diagnostics, 13.","DOI":"10.3390\/diagnostics13040688"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: The convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat. Med."},{"key":"ref_15","unstructured":"OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2024). GPT-4 Technical Report, Cornell University."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"E6","DOI":"10.1038\/sj.bdj.2008.585","article-title":"Unexplained orofacial pain\u2014Is an early diagnosis possible?","volume":"205","author":"Aggarwal","year":"2008","journal-title":"Br. Dent. J."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1016\/j.pain.2004.07.002","article-title":"Can one predict the likely specific orofacial pain syndrome from a self-completed questionnaire?","volume":"111","author":"Macfarlane","year":"2004","journal-title":"Pain"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1111\/head.13689","article-title":"Tooth-Related Pain or Not?","volume":"60","author":"Renton","year":"2020","journal-title":"Headache"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"03331024231187160","DOI":"10.1177\/03331024231187160","article-title":"Orofacial pain for clinicians: A review of constant and attack-like facial pain syndromes","volume":"43","author":"May","year":"2023","journal-title":"Cephalalgia"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1177\/0333102417704605","article-title":"One nerve, three divisions, two professions and nearly no crosstalk?","volume":"37","author":"May","year":"2017","journal-title":"Cephalalgia"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1093\/bja\/aet125","article-title":"Differential diagnosis of facial pain and guidelines for management","volume":"111","author":"Zakrzewska","year":"2013","journal-title":"Br. J. Anaesth."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1016\/j.oooo.2019.02.013","article-title":"Diagnostic delay and suboptimal management in persistent idiopathic facial pain and persistent dentoalveolar pain; a cross-sectional study","volume":"127","author":"Hassona","year":"2019","journal-title":"Oral Surg. Oral Med. Oral Pathol. Oral Radiol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1801","DOI":"10.1016\/j.joen.2021.08.002","article-title":"Referred Orofacial Pain as an Initial Symptom of Distant, Nonmetastatic Cancer: Report of a Case and Review of the Literature","volume":"47","author":"Katchky","year":"2021","journal-title":"J. Endod."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1038\/sj.bdj.2012.468","article-title":"Late diagnosis of an occult tumour\u2014What lessons can we learn?","volume":"212","author":"Parmar","year":"2012","journal-title":"Br. Dent. J."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1038\/s41582-022-00763-1","article-title":"Global epidemiology of migraine and its implications for public health and health policy","volume":"19","author":"Steiner","year":"2023","journal-title":"Nat. Rev. Neurol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1186\/s10194-019-1019-7","article-title":"Trigeminal autonomic cephalalgias presenting in a multidisciplinary tertiary orofacial pain clinic","volume":"20","author":"Wei","year":"2019","journal-title":"J. Headache Pain"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"81","DOI":"10.3109\/00016357.2015.1042040","article-title":"An analysis of dental patient safety incidents in a patient complaint and healthcare supervisory database in Finland","volume":"74","author":"Hiivala","year":"2016","journal-title":"Acta Odontol. Scand."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e540","DOI":"10.1097\/PTS.0000000000000407","article-title":"Classifying Adverse Events in the Dental Office","volume":"17","author":"Kalenderian","year":"2021","journal-title":"J. Patient Saf."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"752","DOI":"10.1111\/j.1468-2982.2008.01586.x","article-title":"The International Classification of Headache Disorders: Accurate Diagnosis of Orofacial Pain?","volume":"28","author":"Benoliel","year":"2008","journal-title":"Cephalalgia"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge","volume":"330","author":"Kanjee","year":"2023","journal-title":"JAMA"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Austin, R., Jones, K., Wright, D., Donaldson, N., and Gallagher, J.E. (2009). Use of the out-of-hours emergency dental service at two south-east London hospitals. BMC Oral Health, 9.","DOI":"10.1186\/1472-6831-9-19"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Raman, S., Ikutame, D., Okura, K., and Matsuka, Y. (2023). Targeted Therapy for Orofacial Pain: A Novel Perspective for Precision Medicine. J. Pers. Med., 13.","DOI":"10.3390\/jpm13030565"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"101648","DOI":"10.1016\/j.jebdp.2021.101648","article-title":"The Voice Of The Patient In Orofacial Pain Management","volume":"22","author":"Lobbezoo","year":"2022","journal-title":"J. Evid.-Based Dent. Pract."},{"key":"ref_34","first-page":"943","article-title":"Building an Automated Orofacial Pain, Headache and Temporomandibular Disorder Diagnosis System","volume":"2020","author":"Nocera","year":"2021","journal-title":"AMIA Annu. Symp. Proc."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1177\/20494637221146854","article-title":"A systematic review of screening diagnostic tools for trigeminal neuralgia","volume":"17","author":"Teshima","year":"2023","journal-title":"Br. J. Pain"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1111\/aas.12697","article-title":"Prevalence and determinants of medication non-adherence in chronic pain patients: A systematic review","volume":"60","author":"Timmerman","year":"2016","journal-title":"Acta Anaesthesiol. Scand."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1535","DOI":"10.1111\/jgh.16561","article-title":"Validation of GPT-4 for clinical event classification: A comparative analysis with ICD codes and human reviewers","volume":"39","author":"Wang","year":"2024","journal-title":"J. Gastroenterol. Hepatol."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Cascella, M., Leoni, M.L.G., Shariff, M.N., and Varrassi, G. (2024). Artificial Intelligence-Driven Diagnostic Processes and Comprehensive Multimodal Models in Pain Medicine. J. Pers. Med., 14.","DOI":"10.20944\/preprints202408.1668.v1"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1097\/j.pain.0000000000001417","article-title":"Machine learning\u2013based prediction of clinical pain using multimodal neuroimaging and autonomic metrics","volume":"160","author":"Lee","year":"2019","journal-title":"Pain"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Palmer, C.E., Marshall, E., Millgate, E., Warren, G., Ewbank, M.P., Cooper, E., Lawes, S., Bouazzaoui, M., Smith, A., and Hutchins-Joss, C. (2024). Combining AI and human support in mental health: A digital intervention with comparable effectiveness to human-delivered care. medRxiv.","DOI":"10.1101\/2024.07.17.24310551"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"f657","DOI":"10.1136\/bmj.f657","article-title":"Features of effective computerised clinical decision support systems: Meta-regression of 162 randomised trials","volume":"346","author":"Roshanov","year":"2013","journal-title":"BMJ"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1038\/s41746-020-0221-y","article-title":"An overview of clinical decision support systems: Benefits, risks, and strategies for success","volume":"3","author":"Sutton","year":"2020","journal-title":"Npj Digit. Med."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1308\/rcsbull.2024.156","article-title":"Five lessons for the debate about health technology and productivity","volume":"106","author":"Horton","year":"2024","journal-title":"Bull. R. Coll. Surg. Engl."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Khanna, N.N., Maindarkar, M.A., Viswanathan, V., Fernandes, J.F.E., Paul, S., Bhagawati, M., Ahluwalia, P., Ruzsa, Z., Sharma, A., and Kolluri, R. (2022). Economics of Artificial Intelligence in Healthcare: Diagnosis vs. Treatment. Healthcare, 10.","DOI":"10.3390\/healthcare10122493"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Lewkowicz, D., Wohlbrandt, A., and Boettinger, E. (2020). Economic impact of clinical decision support interventions based on electronic health records. BMC Health Serv. Res., 20.","DOI":"10.1186\/s12913-020-05688-3"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/s13012-023-01261-8","article-title":"Implementation costs of hospital-based computerised decision support systems: A systematic review","volume":"18","author":"Donovan","year":"2023","journal-title":"Implement. Sci."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"e46089","DOI":"10.2196\/46089","article-title":"Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review","volume":"25","author":"Wang","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_48","first-page":"202","article-title":"Regulation (EU) 2024\/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence and Amending Various Regulations and Directives (Artificial Intelligence Act)","volume":"50","author":"Council","year":"2024","journal-title":"Off. J. Eur. Union L Ser."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/14\/24\/2835\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:53:33Z","timestamp":1760115213000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/14\/24\/2835"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,17]]},"references-count":48,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["diagnostics14242835"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics14242835","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,17]]}}}