{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:56Z","timestamp":1785545036153,"version":"3.56.0"},"reference-count":35,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Systemic lupus erythematosus (SLE) is a complex, fluctuating disease, creating a continuous need for reliable patient information. A prior study concluded that patients with SLE often turn to the internet, including artificial intelligence (AI) chatbots, for information regarding SLE. The rise of AI chatbots as a primary information source presents a critical challenge regarding the accuracy of the information they provide.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the performance of the latest generation of AI chatbots (ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash) in answering frequently asked questions about SLE.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Twenty-two frequently asked questions about SLE in Bahasa Indonesia (the Indonesian language) were posed to each chatbot. Responses were independently and blindly evaluated for accuracy by 5 clinical immunologists using a 4-point Likert scale. Readability was assessed using the Flesch reading ease score formula. Statistical comparisons for accuracy and readability were performed using repeated-measures ANOVA or the Friedman test, followed by the Bonferroni test for pairwise comparisons. The Spearman \u03c1 was used to evaluate correlations among accuracy, readability, and word count.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Gemini 2.5 Flash demonstrated the highest accuracy, with a mean score of 1.25 (SD 0.53), significantly outperforming ChatGPT-4o (mean 1.71, SD 0.61;\n P<\/jats:italic>\n <.001). Gemini 2.5 Flash significantly outperformed ChatGPT-4o in 2 evaluated domains. The interreliability analysis revealed a statistically significant level of agreement among the 5 evaluators across all responses (Kendall\n W<\/jats:italic>\n =0.389;\n P<\/jats:italic>\n <.001). Readability for all 3 chatbots was low (median Flesch reading ease score 42.22\u201046.66). Gemini 2.5 Flash produced the longest responses (8509 total words), followed by DeepSeek-V3 (5410 words) and ChatGPT-4o (3632 words). A significant negative correlation was found between word count and lower accuracy (\u03c1=\u22120.401;\n P<\/jats:italic>\n =.001).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Our study found that ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash provided overall satisfactory responses to SLE-related questions. The highest accuracy was demonstrated by Gemini 2.5 Flash; however, the absolute differences in scores among the 3 AI chatbots were relatively small. All 3 AI chatbots demonstrated low readability, which may limit accessibility for patient use. This finding highlights a critical \u201cblind spot\u201d in which clinical accuracy, as rated by experts, does not equate to patient accessibility. Thus, further research is required to develop more comprehensive evaluation frameworks incorporating safety, factuality, and calibration of AI chatbots across different medical fields and topics.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/89156","type":"journal-article","created":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T16:25:05Z","timestamp":1778257505000},"page":"e89156-e89156","source":"Crossref","is-referenced-by-count":0,"title":["Comparing the Accuracy of ChatGPT-4o, DeepSeek-V3, and Gemini 2.5 Flash in Answering Frequently Asked Questions About Systemic Lupus Erythematosus: Quantitative Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1848-6227","authenticated-orcid":false,"given":"Alvina","family":"Widhani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6951-1480","authenticated-orcid":false,"given":"Suzy","family":"Maria","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3609-6348","authenticated-orcid":false,"given":"Aisha Putri","family":"Chairani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8310-3564","authenticated-orcid":false,"given":"Nabila Yulianingrum Adella","family":"Visco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2322-1365","authenticated-orcid":false,"given":"Muhammad Faiz Amirullah","family":"Nurhadi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0736-0171","authenticated-orcid":false,"given":"Lutfi Airlangga","family":"Harjoprawito","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0227-7735","authenticated-orcid":false,"given":"Dwitya","family":"Elvira","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3308-1463","authenticated-orcid":false,"given":"Irwin","family":"Tedja","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3398-0734","authenticated-orcid":false,"given":"Yuniza","family":"Yuniza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8733-6441","authenticated-orcid":false,"given":"Deasy","family":"Fetarayani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2839-2814","authenticated-orcid":false,"given":"Sukamto","family":"Koesnoe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2718-211X","authenticated-orcid":false,"given":"Bramantya","family":"Wicaksana","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4331-2381","authenticated-orcid":false,"given":"Anshari Saifuddin","family":"Hasibuan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6650-0559","authenticated-orcid":false,"given":"Evy","family":"Yunihastuti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,8]]},"reference":[{"key":"R1","doi-asserted-by":"crossref","unstructured":"Clark M Bailey S . Chatbots in Health Care: Connecting Patients to Information: Emerging Health Technologies. Canadian Agency for Drugs and Technologies in Health; 2024.","DOI":"10.51731\/cjht.2024.818"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1136\/leader-2023-000797","article-title":"ChatGPT and generative AI chatbots: challenges and opportunities for science, medicine and medical leaders","volume":"8","author":"Loh","journal-title":"BMJ Lead"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1136\/ard-2022-223035","article-title":"Global epidemiology of systemic lupus erythematosus: a comprehensive systematic analysis and modelling study","volume":"82","author":"Tian","journal-title":"Ann Rheum Dis"},{"issue":"4","key":"R4","doi-asserted-by":"publisher","DOI":"10.7454\/jpdi.v10i4.1511","article-title":"Survei kebutuhan edukasi pasien lupus eritematosus sistemik [Article in Indonesian]","volume":"10","author":"Josephine","journal-title":"J Penyakit Dalam Indones"},{"key":"R5","unstructured":"Kedaulatan AI untuk memberdayakan Indonesia. Katadata Media Network; 2025. URL: https:\/\/databoks.katadata.co.id\/publikasi\/2025\/01\/23\/kedaulatan-ai-untuk-memberdayakan-indonesia [Accessed 21-05-2026]"},{"key":"R6","unstructured":"AI chatbots vs search engines: 24-month study on traffic trends. OnelittleWeb. URL: https:\/\/onelittleweb.com\/data-studies\/ai-chatbots-vs-search-engines\/ [Accessed 05-12-2025]"},{"key":"R7","unstructured":"Lupus FAQ. Lupus Canada. URL: https:\/\/www.lupuscanada.org\/lupus-faq\/ [Accessed 05-12-2025]"},{"key":"R8","unstructured":"100 questions \u2013 Lupus 100. Lupus Europe. URL: https:\/\/lupus100.org\/en\/100-questions [Accessed 05-12-2025]"},{"issue":"19","key":"R9","doi-asserted-by":"publisher","first-page":"1519","DOI":"10.2106\/JBJS.23.00209","article-title":"Assessing ChatGPT responses to common patient questions regarding total hip arthroplasty","volume":"105","author":"Mika","journal-title":"J Bone Joint Surg Am"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1136\/ard-2023-224762","article-title":"EULAR recommendations for the management of systemic lupus erythematosus: 2023 update","volume":"83","author":"Fanouriakis","journal-title":"Ann Rheum Dis"},{"issue":"9","key":"R11","doi-asserted-by":"publisher","first-page":"1400","DOI":"10.1002\/art.40930","article-title":"2019 European League Against Rheumatism\/American College of Rheumatology classification criteria for systemic lupus erythematosus","volume":"71","author":"Aringer","journal-title":"Arthritis Rheumatol"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","first-page":"720","DOI":"10.1136\/ard-2023-224416","article-title":"EULAR recommendations for the non-pharmacological management of systemic lupus erythematosus and systemic sclerosis","volume":"83","author":"Parodis","journal-title":"Ann Rheum Dis"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1136\/annrheumdis-2021-222020","article-title":"2021 EULAR recommendations regarding lifestyle behaviours and work participation to prevent progression of rheumatic and musculoskeletal diseases","volume":"82","author":"Gwinnutt","journal-title":"Ann Rheum Dis"},{"issue":"3","key":"R14","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick","volume":"32","author":"Flesch","journal-title":"J Appl Psychol"},{"issue":"2","key":"R15","doi-asserted-by":"publisher","first-page":"132","DOI":"10.33369\/diksa.v2i2.3300","article-title":"Analisis tingkat keterbacaan buku siswa bahasa Indonesia kelas VII berbasis kurikulum 2013 [Article in Indonesian]","volume":"2","author":"Gumono","journal-title":"Diksa"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.35473\/po.v5i1","article-title":"Readability of teaching materials by students of Stabn Sriwijaya based on Flesch formula","volume":"5","author":"Suntoro","journal-title":"Philosophica"},{"issue":"2","key":"R17","first-page":"215","volume":"37","author":"Spadaro","journal-title":"Am J Hosp Pharm"},{"issue":"21","key":"R18","first-page":"19","volume":"1","author":"Tomczak","journal-title":"Trends Sport Sci"},{"issue":"5","key":"R19","doi-asserted-by":"publisher","first-page":"2887","DOI":"10.1007\/s00464-024-10739-5","article-title":"Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis","volume":"38","author":"Ghanem","journal-title":"Surg Endosc"},{"issue":"8","key":"R20","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.67996","article-title":"Accuracy and readability of artificial intelligence chatbot responses to vasectomy-related questions: public beware","volume":"16","author":"Carlson","journal-title":"Cureus"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/63677","article-title":"Assessing the quality and reliability of ChatGPT's responses to radiotherapy-related patient queries: comparative study with GPT-3.5 and GPT-4","volume":"11","author":"Grilo","journal-title":"JMIR Cancer"},{"issue":"3","key":"R22","doi-asserted-by":"publisher","first-page":"100094","DOI":"10.1016\/j.cson.2025.100094","article-title":"Assessing the quality and readability of AI chatbot responses to frequently asked questions about basal cell carcinoma","volume":"4","author":"Sidhu","journal-title":"Clin Surg Oncol"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"84","DOI":"10.4103\/1357-6283.210517","article-title":"Assessing reading levels of health information: uses and limitations of Flesch formula","volume":"30","author":"Jindal","journal-title":"Educ Health (Abingdon)"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1038\/s41443-024-00970-y","article-title":"Can AI chatbots accurately answer patient questions regarding vasectomies?","author":"Mouhawasse","journal-title":"Int J Impot Res"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","first-page":"253","DOI":"10.5152\/tud.2025.24098","article-title":"Quality of chatbot responses to the most popular questions regarding erectile dysfunction","volume":"50","author":"Barlas","journal-title":"Urol Res Pract"},{"issue":"10","key":"R26","doi-asserted-by":"publisher","DOI":"10.3390\/jcm14103535","article-title":"The role of artificial intelligence (ChatGPT-4o) in supporting tumor board decisions","volume":"14","author":"Karabu\u011fa","journal-title":"J Clin Med"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1477898","article-title":"Large language models in patient education: a scoping review of applications in medicine","volume":"11","author":"Aydin","journal-title":"Front Med (Lausanne)"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.36922\/aih.4808","article-title":"Role of large language models in improving provider\u2013patient experience and interaction efficiency: a scoping review","volume":"2","author":"Vishwanath","journal-title":"Artif Intell Health"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/77149","article-title":"Leveraging large language models to improve the readability of German online medical texts: evaluation study","volume":"5","author":"Miftaroski","journal-title":"JMIR AI"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.2196\/69955","article-title":"Enhancing the readability of online patient education materials using large language models: cross-sectional study","volume":"27","author":"Will","journal-title":"J Med Internet Res"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1016\/j.hsr.2025.100249","article-title":"Addressing language barriers in U.S. healthcare: the role of CLAS standards, telehealth, and policy in supporting limited English proficiency populations","volume":"17","author":"Wong","journal-title":"Health Sci Rev"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.37349\/edht.2024.00006","article-title":"Warning: artificial intelligence chatbots can generate inaccurate medical and scientific information and references","volume":"2","author":"Clelland","journal-title":"Explor Digit Health Technol"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjao.2024.100296","article-title":"A comparative study of English and Japanese ChatGPT responses to anaesthesia-related medical questions","volume":"10","author":"Ando","journal-title":"BJA Open"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs","volume":"7","author":"Wang","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-02601-y","article-title":"Large language model evaluation in autoimmune disease clinical questions comparing ChatGPT 4o, Claude 3.5 Sonnet and Gemini 1.5 pro","volume":"15","author":"Ma","journal-title":"Sci Rep"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e89156","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:24:43Z","timestamp":1785536683000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e89156"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,8]]},"references-count":35,"URL":"https:\/\/doi.org\/10.2196\/89156","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,8]]},"article-number":"v10i1e89156"}}