{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T14:12:49Z","timestamp":1791987169855,"version":"4.4.0"},"reference-count":32,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T00:00:00Z","timestamp":1787875200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: To compare the diagnostic performance of Claude Opus 4.7 and Gemini Pro 3 for the differential diagnosis of oral autoimmune blistering diseases (AIBDs) and evaluate their diagnostic reasoning, confidence, and calibration. Materials and Methods: This retrospective multicenter paired diagnostic accuracy study included 200 clinicopathologically confirmed AIBD cases (50 each of pemphigus vulgaris, mucous membrane pemphigoid, bullous pemphigoid, and linear IgA bullous dermatosis). Each case was independently assessed by both models using identical standardized clinical information and clinical photographic inputs. The task required forced-choice classification among the four predefined diseases. Histopathological and direct immunofluorescence findings were used exclusively to establish the clinicopathological reference diagnosis and were not provided to the AI models. The reference diagnosis was established by clinicopathological correlation. The primary outcome was diagnostic accuracy. Secondary outcomes included disease-specific diagnostic performance, Cohen\u2019s \u03ba, ROC analysis, calibration, confidence, reasoning quality, management recommendations, and error patterns. Pre-consensus inter-rater reliability of the two human assessors was also evaluated using Cohen\u2019s \u03ba for binary outcomes and weighted Cohen\u2019s \u03ba for the ordinal reasoning-quality score. Results: Claude achieved significantly higher diagnostic accuracy than Gemini (92.0% vs. 86.0%, p = 0.012), stronger agreement with the reference standard (\u03ba = 0.893 vs. 0.813), and superior discrimination (macro-AUC 0.998 vs. 0.965). Claude demonstrated higher key diagnostic-feature identification (92.0% vs. 86.0%; p = 0.012) and higher clinical-reasoning scores (61.0% vs. 40.0% of responses rated good; Wilcoxon p &lt; 0.001; r = 0.47), whereas management recommendations did not differ significantly (100.0% vs. 98.0%; p = 0.125). Calibration results were metric-dependent: Claude had a lower one-vs-rest Brier score (0.0468 vs. 0.0558), whereas Gemini had a lower expected calibration error (0.083 vs. 0.251). For both models, the predominant error was misclassification of linear IgA bullous dermatosis as mucous membrane pemphigoid. Conclusions: Both multimodal LLMs showed high performance in this controlled four-class benchmark, with Claude Opus 4.7 outperforming Gemini Pro 3 in overall accuracy and reasoning quality. However, these findings do not establish autonomous diagnostic capability, clinical effectiveness, or safety. The LABD\u2013MMP misclassification and metric-dependent calibration highlight important limitations. The models should therefore be regarded as investigational adjunctive decision-support tools requiring clinician oversight and diagnostic verification. Prospective external and human-in-the-loop validation is required before clinical implementation.<\/jats:p>","DOI":"10.3390\/diagnostics16172761","type":"journal-article","created":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T12:27:46Z","timestamp":1787920066000},"page":"2761","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Diagnostic Accuracy of Multimodal Large Language Models for Four-Class Benchmark of Oral Autoimmune Blistering Diseases: A Multicenter Paired Study"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5069-8257","authenticated-orcid":false,"given":"Asmaa","family":"Abou-Bakr","sequence":"first","affiliation":[{"name":"Oral Medicine and Periodontology, Faculty of Dentistry, Galala University, Suez 43511, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"methodology","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"}]},{"given":"Salma M","family":"Saad","sequence":"additional","affiliation":[{"name":"Oral Medicine and Periodontology Department, Faculty of Dentistry, Cairo University, Cairo 11553, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"resources","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"}]},{"given":"Nevine H. Kheir El","family":"Din","sequence":"additional","affiliation":[{"name":"Oral Medicine and Periodontology, Ain Shams University, Cairo 11566, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"supervision","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8535-5876","authenticated-orcid":false,"given":"Abdullah Bin","family":"Nabhan","sequence":"additional","affiliation":[{"name":"Oral Medicine and Orofacial Pain, College of Dentistry, Prince Sattam Bin Abdulaziz University, Al-Kharj 11942, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"investigation","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"}]},{"given":"Amal","family":"Bajonaid","sequence":"additional","affiliation":[{"name":"Oral Medicine Department of Maxillofacial Surgery and Diagnostic Sciences, College of Dentistry, Jazan University, Jazan 45142, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"}]},{"given":"Asma Saleh","family":"Almeslet","sequence":"additional","affiliation":[{"name":"Oral and Maxillofacial Surgery and Diagnostic Sciences, College of Medicine and Dentistry, Riyadh Elm University, Riyadh 12734, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-8010-9265","authenticated-orcid":false,"given":"Fatma E. A.","family":"Hassanein","sequence":"additional","affiliation":[{"name":"Oral Medicine, Periodontology, and Oral Diagnosis, Faculty of Dentistry, King Salman International University, South Sinai 46612, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,8,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Witte, M., Zillikens, D., and Schmidt, E. (2018). Diagnosis of Autoimmune Blistering Diseases. Front. Med., 5.","DOI":"10.3389\/fmed.2018.00296"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1016\/j.mpmed.2025.05.008","article-title":"Autoimmune blistering skin diseases","volume":"53","author":"Fletcher","year":"2025","journal-title":"Medicine"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1038\/s41415-025-8666-7","article-title":"The potential of artificial intelligence in the early detection of systemic diseases during routine dental care","volume":"239","author":"Veseli","year":"2025","journal-title":"Br. Dent. J."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1001\/jamadermatol.2018.4390","article-title":"Assessment of Diagnostic Strategy for Early Recognition of Bullous and Nonbullous Variants of Pemphigoid","volume":"155","author":"Meijer","year":"2019","journal-title":"JAMA Dermatol."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"van Beek, N., Holtsche, M.M., Atefi, I., Olbrich, H., Schmitz, M.J., Pruessmann, J., Vorobyev, A., and Schmidt, E. (2024). State-of-the-art diagnosis of autoimmune blistering diseases. Front. Immunol., 15.","DOI":"10.3389\/fimmu.2024.1363032"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1111\/prd.12263","article-title":"Oral autoimmune vesicobullous diseases: Classification, clinical presentations, molecular mechanisms, diagnostic algorithms, and management","volume":"80","author":"Leuci","year":"2019","journal-title":"Periodontol. 2000"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1685","DOI":"10.1111\/jdv.15679","article-title":"Review of autoimmune blistering diseases: The Pemphigoid diseases","volume":"33","author":"Daniel","year":"2019","journal-title":"J. Eur. Acad. Dermatol. Venereol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hassanein, F.E., Alkabazi, M., Tassoker, M., Ahmed, Y., Alsaeed, S., and Abou-Bakr, A. (2026). Multimodal large language models for oral lesion diagnosis: A systematic review of diagnostic performance and clinical utility. Front. Oral Health, 7.","DOI":"10.3389\/froh.2026.1748450"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1915","DOI":"10.1007\/s00405-025-09925-5","article-title":"Comparative diagnostic accuracy of ChatGPT models in salivary gland disease: A multimodal vignette-based evaluation","volume":"283","author":"Eissa","year":"2026","journal-title":"Eur. Arch. Oto-Rhino-Laryngol."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Han, S.S., Park, G.H., Lim, W., Kim, M.S., Na, J.I., Park, I., and Chang, S.E. (2018). Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: Automatic construction of onychomycosis datasets by region-based convolutional deep neural network. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0191493"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1038\/s41591-020-0842-3","article-title":"A deep learning system for differential diagnosis of skin diseases","volume":"26","author":"Liu","year":"2020","journal-title":"Nat. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","year":"2023","journal-title":"Nature"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N. Engl. J. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"h5527","DOI":"10.1136\/bmj.h5527","article-title":"STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies","volume":"351","author":"Bossuyt","year":"2015","journal-title":"BMJ"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3283","DOI":"10.1038\/s41591-025-03953-8","article-title":"The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence","volume":"31","author":"Sounderajah","year":"2025","journal-title":"Nat. Med."},{"key":"ref_16","first-page":"e92089","article-title":"Comparative Analysis of Large Language Models in Dermatological Diagnosis: An Evaluation of Diagnostic Accuracy","volume":"17","author":"Tekchandani","year":"2025","journal-title":"Cureus"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5649","DOI":"10.1038\/s41467-024-50043-3","article-title":"Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4","volume":"15","author":"Zhou","year":"2024","journal-title":"Nat. Commun."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1515\/dx-2025-0014","article-title":"Large language models for dermatological image interpretation\u2014A comparative study","volume":"13","author":"Cirkel","year":"2026","journal-title":"Diagnosis"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3344","DOI":"10.1111\/odi.70009","article-title":"Diagnostic Performance of Multimodal Large Language Models in the Analysis of Oral Pathology","volume":"31","author":"Freire","year":"2025","journal-title":"Oral Dis."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"18633","DOI":"10.1007\/s00521-025-11394-0","article-title":"Autoimmune blistering skin disease classification: Classical machine learning, deep learning, and hybrid approaches","volume":"37","author":"Singh","year":"2025","journal-title":"Neural Comput. Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1016\/S1076-6332(03)80329-X","article-title":"Confidence bounds when the estimated ROC area is 1.01","volume":"9","author":"Obuchowski","year":"2002","journal-title":"Acad. Radiol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"4230","DOI":"10.1002\/sim.9025","article-title":"Minimum sample size for external validation of a clinical prediction model with a binary outcome","volume":"40","author":"Riley","year":"2021","journal-title":"Stat. Med."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1161\/CIRCULATIONAHA.114.014508","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. The TRIPOD Group","volume":"131","author":"Collins","year":"2015","journal-title":"Circulation"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e074820","DOI":"10.1136\/bmj-2023-074820","article-title":"Evaluation of clinical prediction models (part 2): How to undertake an external validation study","volume":"384","author":"Riley","year":"2024","journal-title":"BMJ"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mee, J.B. (2023). Diagnostic Techniques in Autoimmune Blistering Diseases. Br. J. Biomed. Sci., 80.","DOI":"10.3389\/bjbs.2023.11809"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1016\/S0140-6736(12)61140-4","article-title":"Pemphigoid diseases","volume":"381","author":"Schmidt","year":"2013","journal-title":"Lancet"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1515\/dx-2020-0020","article-title":"Teamwork in clinical reasoning\u2013cooperative or parallel play?","volume":"7","author":"Olson","year":"2020","journal-title":"Diagnosis"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"109357","DOI":"10.1016\/j.identj.2025.109357","article-title":"A Multimodal Large Language Model Framework for Clinical Subtyping and Malignant Transformation Risk Prediction in Oral Lichen Planus: A Paired Comparison with Expert Clinicians","volume":"76","author":"Robaian","year":"2026","journal-title":"Int. Dent. J."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1038\/s41746-025-02015-0","article-title":"Artificial intelligence for autoimmune diseases","volume":"8","author":"Mahajan","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1016\/j.clindermatol.2024.06.008","article-title":"Artificial intelligence in autoimmune bullous dermatoses","volume":"42","author":"Manuelyan","year":"2024","journal-title":"Clin. Dermatol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2268","DOI":"10.1111\/jdv.20031","article-title":"Human-multimodal deep learning collaboration in \u2018precise\u2019 diagnosis of lupus erythematosus subtypes and similar skin diseases","volume":"38","author":"Li","year":"2024","journal-title":"J. Eur. Acad. Dermatol. Venereol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Yang, Y., Liu, Y., Chen, Y., Luo, D., Xu, K., and Zhang, L. (2024). Artificial intelligence for predicting treatment responses in autoimmune rheumatic diseases: Advancements, challenges, and future perspectives. Front. Immunol., 15.","DOI":"10.3389\/fimmu.2024.1477130"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/17\/2761\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T12:39:04Z","timestamp":1787920744000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/17\/2761"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,28]]},"references-count":32,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["diagnostics16172761"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16172761","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,28]]}}}