{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T21:49:11Z","timestamp":1784238551441,"version":"3.55.0"},"reference-count":43,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,3,31]],"date-time":"2025-03-31T00:00:00Z","timestamp":1743379200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>Kawasaki disease (KD) presents complex clinical challenges in diagnosis, treatment, and long-term management, requiring a comprehensive understanding by both parents and healthcare providers. With advancements in artificial intelligence (AI), large language models (LLMs) have shown promise in supporting medical practice. This study aims to evaluate and compare the appropriateness and comprehensibility of different LLMs in answering clinically relevant questions about KD and assess the impact of different prompting strategies.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Twenty-five questions were formulated, incorporating three prompting strategies: No prompting (NO), Parent-friendly (PF), and Doctor-level (DL). These questions were input into three LLMs: ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Responses were evaluated based on appropriateness, educational quality, comprehensibility, cautionary statements, references, and potential misinformation, using Information Quality Grade, Global Quality Scale (GQS), Flesch Reading Ease (FRE) score, and word count.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Significant differences were found among the LLMs in terms of response educational quality, accuracy, and comprehensibility (<jats:italic>p<\/jats:italic>\u202f&amp;lt;\u202f0.001). Claude 3.5 provided the highest proportion of completely correct responses (51.1%) and achieved the highest median GQS score (5.0), outperforming GPT-4o (4.0) and Gemini 1.5 (3.0) significantly. Gemini 1.5 achieved the highest FRE score (31.5) and provided highest proportion of responses assessed as comprehensible (80.4%). Prompting strategies significantly affected LLM responses. Claude 3.5 Sonnet with DL prompting had the highest completely correct rate (81.3%), while PF prompting yielded the most acceptable responses (97.3%). Gemini 1.5 Pro showed minimal variation across prompts but excelled in comprehensibility (98.7% under PF prompting).<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>This study indicates that LLMs have great potential in providing information about KD, but their use requires caution due to quality inconsistencies and misinformation risks. Significant discrepancies existed across LLMs and prompting strategies. Claude 3.5 Sonnet offered the best response quality and accuracy, while Gemini 1.5 Pro excelled in comprehensibility. PF prompting with Claude 3.5 Sonnet is most recommended for parents seeking KD information. As AI evolves, expanding research and refining models is crucial to ensure reliable, high-quality information.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2025.1571503","type":"journal-article","created":{"date-parts":[[2025,3,31]],"date-time":"2025-03-31T05:22:17Z","timestamp":1743398537000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Assessing large language models as assistive tools in medical consultations for Kawasaki disease"],"prefix":"10.3389","volume":"8","author":[{"given":"Chunyi","family":"Yan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zexi","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yongzhou","family":"Liang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuran","family":"Shao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fan","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nanjun","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bowen","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chuan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaiyu","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,3,31]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"1353","DOI":"10.1227\/neu.0000000000002632","article-title":"Performance of ChatGPT and GPT-4 on neurosurgery written board examinations","volume":"93","author":"Ali","year":"2023","journal-title":"Neurosurgery"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1186\/s12969-021-00566-6","article-title":"The occurrence of coronary artery lesions in Kawasaki disease based on C-reactive protein levels: a retrospective cohort study","volume":"19","author":"An","year":"2021","journal-title":"Pediatr. Rheumatol. Online J."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"e39238","DOI":"10.7759\/cureus.39238","article-title":"High rates of fabricated and inaccurate references in ChatGPT-generated medical content","volume":"15","author":"Bhattacharyya","year":"2023","journal-title":"Cureus"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s11255-023-03773-0","article-title":"Evaluating the performance of ChatGPT in answering questions related to urolithiasis","volume":"56","author":"Cakir","year":"2024","journal-title":"Int. Urol. Nephrol."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"1884","DOI":"10.1007\/s11606-019-05109-0","article-title":"Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet","volume":"34","author":"Daraz","year":"2019","journal-title":"J. Gen. Intern. Med."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"1168","DOI":"10.1007\/s11604-024-01606-3","article-title":"Evaluating ChatGPT-4V in chest CT diagnostics: a critical image interpretation assessment","volume":"42","author":"Dehdab","year":"2024","journal-title":"Jpn. J. Radiol."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"2086","DOI":"10.1007\/s00415-023-12168-1","article-title":"ChatGPT fails challenging the recent ESCMID brain abscess guideline","volume":"271","author":"Dyckhoff-Shen","year":"2024","journal-title":"J. Neurol."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"9","DOI":"10.14740\/cr993","article-title":"Kawasaki disease: global burden and genetic background","volume":"11","author":"Elakabawi","year":"2020","journal-title":"Cardiol. Res."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1038\/s41568-021-00399-1","article-title":"Artificial intelligence in cancer research, diagnosis and therapy","volume":"21","author":"Elemento","year":"2021","journal-title":"Nat. Rev. Cancer"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1177\/0033354919874074","article-title":"Online health information seeking among US adults: measuring Progress toward a healthy people 2020 objective","volume":"134","author":"Finney Rutten","year":"2019","journal-title":"Public Health Rep. (Washington, DC: 1974)"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"561","DOI":"10.5005\/jp-journals-10071-24725","article-title":"End-of-life care patient information leaflets-a comparative evaluation of artificial intelligence-generated content for readability, sentiment, accuracy, completeness, and suitability: ChatGPT vs Google Gemini","volume":"28","author":"Gondode","year":"2024","journal-title":"Indian J. Crit. Care Med."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1016\/j.ajem.2024.03.017","article-title":"Comparison of emergency medicine specialist, cardiologist, and chat-GPT in electrocardiography assessment","volume":"80","author":"G\u00fcnay","year":"2024","journal-title":"Am. J. Emerg. Med."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1056\/NEJMra2302038","article-title":"Artificial intelligence and machine learning in clinical medicine, 2023","volume":"388","author":"Haug","year":"2023","journal-title":"N. Engl. J. Med."},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1093\/europace\/euad369","article-title":"Accuracy and comprehensibility of chat-based artificial intelligence for patient information on atrial fibrillation and cardiac implantable electronic devices","volume":"26","author":"Hillmann","year":"2023","journal-title":"Europace"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1093\/jncics\/pkad010","article-title":"Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift","volume":"7","author":"Hopkins","year":"2023","journal-title":"JNCI Cancer Spectr."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"108321","DOI":"10.1016\/j.envint.2023.108321","article-title":"The association between prenatal per-and polyfluoroalkyl substance levels and Kawasaki disease among children of up to 4 years of age: a prospective birth cohort of the Japan environment and Children's study","volume":"183","author":"Iwata","year":"2024","journal-title":"Environ. Int."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"84","DOI":"10.4103\/1357-6283.210517","article-title":"Assessing reading levels of health information: uses and limitations of Flesch formula","volume":"30","author":"Jindal","year":"2017","journal-title":"Educ. Health (Abingdon)"},{"key":"ref18","doi-asserted-by":"publisher","first-page":"892","DOI":"10.1016\/S0022-3476(75)80220-4","article-title":"Coronary aneurysms in infants and young children with acute febrile mucocutaneous lymph node syndrome","volume":"86","author":"Kato","year":"1975","journal-title":"J. Pediatr."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1161\/01.CIR.94.6.1379","article-title":"Long-term consequences of Kawasaki disease. A 10- to 21-year follow-up study of 594 patients","volume":"94","author":"Kato","year":"1996","journal-title":"Circulation"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1093\/postmj\/qgae065","article-title":"ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review","volume":"100","author":"K\u0131yak","year":"2024","journal-title":"Postgrad. Med. J."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J. Chiropr. Med."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"1519","DOI":"10.1002\/ohn.759","article-title":"Performance and consistency of ChatGPT-4 versus otolaryngologists: a clinical case series","volume":"170","author":"Lechien","year":"2024","journal-title":"Otolaryngology"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"e59273","DOI":"10.2196\/59273","article-title":"Claude 3 opus and ChatGPT with GPT-4 in dermoscopic image analysis for melanoma diagnosis: comparative performance analysis","volume":"12","author":"Liu","year":"2024","journal-title":"JMIR Med. Inform."},{"key":"ref24","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.5435\/JAAOS-D-23-00396","article-title":"Comparison of ChatGPT-3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment examinations","volume":"31","author":"Massey","year":"2023","journal-title":"J. Am. Acad. Orthop. Surg."},{"key":"ref25","doi-asserted-by":"publisher","first-page":"e927","DOI":"10.1161\/CIR.0000000000000484","article-title":"Diagnosis, treatment, and long-term Management of Kawasaki Disease: a scientific statement for health professionals from the American Heart Association","volume":"135","author":"McCrindle","year":"2017","journal-title":"Circulation"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1007\/s00266-024-04343-0","article-title":"Evaluation of rhinoplasty information from ChatGPT, Gemini, and Claude for readability and accuracy","author":"Meyer","year":"2024","journal-title":"Aesth. Plast. Surg."},{"key":"ref27","doi-asserted-by":"publisher","first-page":"216","DOI":"10.2188\/jea.JE20110126","article-title":"Epidemiologic features of Kawasaki disease in Japan: results of the 2009-2010 nationwide survey","volume":"22","author":"Nakamura","year":"2012","journal-title":"J. Epidemiol."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"1738","DOI":"10.1016\/j.jacc.2015.12.073","article-title":"Kawasaki Disease","volume":"67","author":"Newburger","year":"2016","journal-title":"J. Am. Coll. Cardiol."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1038\/s41598-023-50884-w","article-title":"Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy","volume":"14","author":"Onder","year":"2024","journal-title":"Sci. Rep."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1016\/j.clgc.2023.12.017","article-title":"Urological cancers and ChatGPT: assessing the quality of information and possible risks for patients","volume":"22","author":"Ozgor","year":"2024","journal-title":"Clin. Genitourin. Cancer"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"5304","DOI":"10.12998\/wjcc.v12.i23.5304","article-title":"Application of artificial intelligence in the diagnosis and treatment of Kawasaki disease","volume":"12","author":"Pan","year":"2024","journal-title":"World J. Clin. Cases"},{"key":"ref32","doi-asserted-by":"publisher","first-page":"756","DOI":"10.1111\/bjhp.12571","article-title":"Measuring online health-seeking behaviour: construction and initial validation of a new scale","volume":"27","author":"Popovac","year":"2022","journal-title":"Br. J. Health Psychol."},{"key":"ref33","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1038\/s41591-021-01614-0","article-title":"AI in health and medicine","volume":"28","author":"Rajpurkar","year":"2022","journal-title":"Nat. Med."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"6099","DOI":"10.1007\/s00405-024-08828-1","article-title":"Assessing the use of the novel tool Claude 3 in comparison to ChatGPT 4.0 as an artificial intelligence tool in the diagnosis and therapy of primary head and neck cancer cases","volume":"281","author":"Schmidl","year":"2024","journal-title":"Eur. Arch. Otorhinolaryngol."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"99","DOI":"10.3390\/info15020099","article-title":"Generative pre-trained transformer (GPT) in research: a systematic review on data augmentation","volume":"15","author":"Sufi","year":"2024","journal-title":"Information"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"e59960","DOI":"10.7759\/cureus.59960","article-title":"Assessing the responses of large language models (ChatGPT-4, Gemini, and Microsoft copilot) to frequently asked questions in breast imaging: a study on readability and accuracy","volume":"16","author":"Tepe","year":"2024","journal-title":"Cureus"},{"key":"ref39","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat. Med."},{"key":"ref40","doi-asserted-by":"publisher","first-page":"14045","DOI":"10.1038\/s41598-023-41032-5","article-title":"Fabrication and errors in the bibliographic citations generated by ChatGPT","volume":"13","author":"Walters","year":"2023","journal-title":"Sci. Rep."},{"key":"ref41","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs","volume":"7","author":"Wang","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref42","doi-asserted-by":"publisher","first-page":"e56426","DOI":"10.2196\/56426","article-title":"Assessing ChatGPT as a medical consultation assistant for chronic hepatitis B: cross-language study of English and Chinese","volume":"12","author":"Wang","year":"2024","journal-title":"JMIR Med. Inform."},{"key":"ref43","doi-asserted-by":"publisher","first-page":"3917","DOI":"10.2147\/JMDH.S473680","article-title":"Comparison of the performance of ChatGPT, Claude and Bard in support of myopia prevention and control","volume":"17","author":"Wang","year":"2024","journal-title":"J. Multidiscip. Healthc."},{"key":"ref44","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1097\/01.inf.0000183786.70519.fa","article-title":"Kawasaki disease: infection, immunity and genetics","volume":"24","author":"Wang","year":"2005","journal-title":"Pediatr. Infect. Dis. J."},{"key":"ref45","doi-asserted-by":"publisher","first-page":"1184","DOI":"10.1016\/j.arth.2024.01.029","article-title":"Chat generative Pretrained transformer (ChatGPT) and bard: artificial intelligence does not yet provide clinically supported answers for hip and knee osteoarthritis","volume":"39","author":"Yang","year":"2024","journal-title":"J. Arthroplast."}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1571503\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,31]],"date-time":"2025-03-31T05:22:19Z","timestamp":1743398539000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1571503\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,31]]},"references-count":43,"alternative-id":["10.3389\/frai.2025.1571503"],"URL":"https:\/\/doi.org\/10.3389\/frai.2025.1571503","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,31]]},"article-number":"1571503"}}