{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T01:22:52Z","timestamp":1791249772918,"version":"4.1.0"},"reference-count":40,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T00:00:00Z","timestamp":1787097600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Public Health"],"abstract":"\n Objective<\/jats:title>\n To systematically evaluate the quality and readability of health information generated by four large language models (LLMs) in response to inquiries regarding type 2 diabetes mellitus (T2DM), using an authoritative Chinese clinical guideline as the reference standard.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A total of 124 standardized questions were extracted from the Chinese Type 2 Diabetes Popular Science Guidelines. Six endocrinologists and diabetes specialists conducted independent, blind evaluations using the CLEAR tool (Completeness, Lack of false Information, Evidence, Appropriateness, Relevance) and PEMAT-P (Patient Education Materials Assessment Tool for Printable materials). Response characteristics were also recorded. Between-model differences were tested using the Kruskal\u2013Wallis H test with Bonferroni pairwise comparisons.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n All four models achieved total CLEAR scores within the \u201cvery good\u201d range (19\u201325), with no significant differences seen between models (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f1.985,\n p<\/jats:italic>\n \u202f=\u202f0.576). No significant differences were observed in the dimensions of Lack of false information (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f7.644,\n p<\/jats:italic>\n \u202f=\u202f0.054), Evidence (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f2.309,\n p<\/jats:italic>\n \u202f=\u202f0.511), and Relevance (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f7.516,\n p<\/jats:italic>\n \u202f=\u202f0.057). However, significant differences emerged in Completeness (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f47.661,\n p<\/jats:italic>\n \u202f&lt;\u202f0.001) and Appropriateness (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f88.360,\n p<\/jats:italic>\n \u202f&lt;\u202f0.001). Claude-4.0 received the lowest score in Completeness (median 4.00, IQR 3.00\u20135.00) but achieved the highest ranking in Appropriateness (median 4.00, IQR 4.00\u20135.00). On the PEMAT-P, understandability differed significantly across models (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f159.120,\n p<\/jats:italic>\n \u202f&lt;\u202f0.001), yet all models surpassed the 70% threshold, with ChatGPT-4.1 highest (median 91.91%, IQR 91.91\u2013100.00%). However, despite significant differences among the various models (\u03c7\n 2<\/jats:sup>\n \u202f=\u202f354.023,\n p<\/jats:italic>\n \u202f&lt;\u202f0.001), only ERNIE Bot 4.5 Turbo (median 75.00%, IQR75.00\u201375.00%) surpassed the 70% threshold, with no single model demonstrating consistent superiority across all dimensions.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n Although the four LLMs generally provide accurate and pertinent information regarding type 2 diabetes, enduring limits in actionability and inconsistencies among models in content completeness and understandability restrict their effective use in diabetic patient education. Future development should prioritize stronger step-by-step behavioral guidance and differentiated, scenario-specific model deployment to enhance their value in patient-facing diabetes self-management support.<\/jats:p>\n <\/jats:sec>","DOI":"10.3389\/fpubh.2026.1900281","type":"journal-article","created":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T15:37:06Z","timestamp":1787153826000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo"],"prefix":"10.3389","volume":"14","author":[{"given":"Zhaoxia","family":"Huang","sequence":"first","affiliation":[{"name":"Department of Ophthalmology, The Affiliated Hospital of Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuxin","family":"Bai","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junyue","family":"Luo","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zihang","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunlan","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyu","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liying","family":"Ruan","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiaxin","family":"Zeng","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qianhui","family":"Jia","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luyao","family":"Deng","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meifeng","family":"Xiong","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziyi","family":"Qi","sequence":"additional","affiliation":[{"name":"School of Nursing, Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenxi","family":"Lyu","sequence":"additional","affiliation":[{"name":"College of Literature, Yancheng Teachers University","place":["Yancheng, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Ophthalmology, The Affiliated Hospital of Southwest Medical University","place":["Luzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,8,19]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"4613","DOI":"10.3390\/nu15214613","article-title":"The global burden of type 2 diabetes attributable to dietary risks: insights from the global burden of disease study 2019","volume":"15","author":"Forray","year":"2023","journal-title":"Nutrients"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1038\/s41591-025-04027-5","article-title":"The global macroeconomic burden of diabetes mellitus","volume":"32","author":"Chen","year":"2026","journal-title":"Nat Med"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"e72167","DOI":"10.2196\/72167","article-title":"Application of digital tools in the Care of Patients with diabetes: scoping review","volume":"27","author":"Yan","year":"2025","journal-title":"J Med Internet Res"},{"key":"ref4","doi-asserted-by":"publisher","first-page":"6856","DOI":"10.3390\/ijerph17186856","article-title":"Social and demographic patterns of health-related internet use among adults in the United States: a secondary data analysis of the health information National Trends Survey","volume":"17","author":"Calixte","year":"2020","journal-title":"Int J Environ Res Public Health"},{"key":"ref5","doi-asserted-by":"publisher","first-page":"e64290","DOI":"10.2196\/64290","article-title":"Laypeople's use of and attitudes toward large language models and search Engines for Health Queries: survey study","volume":"27","author":"Mendel","year":"2025","journal-title":"J Med Internet Res"},{"key":"ref6","doi-asserted-by":"publisher","first-page":"2840","DOI":"10.1038\/s41598-024-52853-3","article-title":"Evaluating AI in medicine: a comparative analysis of expert and ChatGPT responses to colorectal cancer questions","volume":"14","author":"Peng","year":"2024","journal-title":"Sci Rep"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1186\/s12903-025-06267-w","article-title":"Comparative evaluation of responses from Deepseek-R1, Chatgpt-O1, Chatgpt-4, and dental Gpt Chatbots to patient inquiries about dental and maxillofacial prostheses","volume":"25","author":"\u00d6zcivelek","year":"2025","journal-title":"BMC Oral Health"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"105871","DOI":"10.1016\/j.ijmedinf.2025.105871","article-title":"Evaluating Ai-generated patient education materials for spinal surgeries: comparative analysis of readability and Discern quality across Chatgpt and Deepseek models","volume":"198","author":"Zhou","year":"2025","journal-title":"Int J Med Inform"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"8996","DOI":"10.1177\/2055207620948996","article-title":"Factors affecting the quality and reliability of online health information","volume":"6","author":"Battineni","year":"2020","journal-title":"Digit Health"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1136\/bmj.h3480","article-title":"Evaluation of symptom checkers for self diagnosis and triage: audit study","volume":"351","author":"Semigran","year":"2015","journal-title":"BMJ"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/s2589-7500(23)00225-x","article-title":"Assessing the potential of Gpt-4 to perpetuate racial and gender biases in health care: a model evaluation study","volume":"6","author":"Zack","year":"2024","journal-title":"Lancet Digit Health"},{"key":"ref12","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence Chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1089\/end.2023.0696","article-title":"Evaluation of the current status of artificial intelligence for Endourology patient education: a blind comparison of Chatgpt and Google bard against traditional information resources","volume":"38","author":"Connors","year":"2024","journal-title":"J Endourol"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1186\/s12911-025-03202-5","article-title":"Can large language models follow guidelines? A comparative study of Chatgpt-4o and Deepseek Ai in clavicle fracture management based on Aaos recommendations","volume":"25","author":"Ke\u00e7eci","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"101071","DOI":"10.1016\/j.waojou.2025.101071","article-title":"How accurate are ChatGPT-4 responses in chronic urticaria? A critical analysis with information quality metrics","volume":"18","author":"Cherrez-Ojeda","year":"2025","journal-title":"World Allergy Organ J"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1941","DOI":"10.1097\/js9.0000000000001066","article-title":"Evaluation of large language models in breast cancer clinical scenarios: a comparative analysis based on ChatGPT-3.5, ChatGPT-4.0, and Claude2","volume":"110","author":"Deng","year":"2024","journal-title":"Int J Surg"},{"key":"ref17","doi-asserted-by":"publisher","first-page":"e65083","DOI":"10.7759\/cureus.65083","article-title":"Large language models take on cardiothoracic surgery: a comparative analysis of the performance of four models on American Board of Thoracic Surgery Exam Questions in 2023","volume":"16","author":"Khalpey","year":"2024","journal-title":"Cureus."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"e101006","DOI":"10.1136\/bmjhci-2023-101006","article-title":"Assessment of the information provided by Chatgpt regarding exercise for patients with type 2 diabetes: a pilot study","volume":"31","author":"Chung","year":"2024","journal-title":"BMJ Health Care Inform"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"S684","DOI":"10.4103\/ijo.Ijo_2510_23","article-title":"Assessing the appropriateness and completeness of Chatgpt-4\u2019s Ai-generated responses for queries related to diabetic retinopathy","volume":"72","author":"Subramanian","year":"2024","journal-title":"Indian J Ophthalmol"},{"key":"ref20","doi-asserted-by":"publisher","first-page":"857","DOI":"10.2147\/amep.S479801","article-title":"Human versus artificial intelligence: Chatgpt-4 outperforming Bing, bard, Chatgpt-3.5 and humans in clinical chemistry multiple-choice questions","volume":"15","author":"Sallam","year":"2024","journal-title":"Adv Med Educ Pract"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"e69996","DOI":"10.7759\/cureus.69996","article-title":"Assessing the quality of patient education materials on cardiac catheterization from artificial intelligence chatbots: an observational cross-sectional study","volume":"16","author":"Behers","year":"2024","journal-title":"Cureus"},{"key":"ref22","doi-asserted-by":"publisher","first-page":"e2530220","DOI":"10.1001\/jamanetworkopen.2025.30220","article-title":"Reporting guideline for chatbot health advice studies","volume":"8","author":"Huo","year":"2025","journal-title":"JAMA Netw Open"},{"key":"ref23","volume-title":"Chinese Type 2 Diabetes Popular Science Guidelines","author":"Zhu","year":"2022"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"104668","DOI":"10.1016\/j.nepr.2025.104668","article-title":"How large language model responds to common depression questions: a comparative analysis of Chatgpt-4.0, Deepseek, Google Gemini and perplexity","volume":"90","author":"You","year":"2026","journal-title":"Nurse Educ Pract"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"e49373","DOI":"10.7759\/cureus.49373","article-title":"Pilot testing of a tool to standardize the assessment of the quality of health information generated by artificial intelligence-based models","volume":"15","author":"Sallam","year":"2023","journal-title":"Cureus."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1016\/j.pec.2014.05.027","article-title":"Development of the patient education materials assessment tool (Pemat): a new measure of understandability and Actionability for print and audiovisual patient information","volume":"96","author":"Shoemaker","year":"2014","journal-title":"Patient Educ Couns"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"e97155","DOI":"10.7759\/cureus.97155","article-title":"Can large language models draft safe, reliable patient leaflets on driving after stroke in comparison to the Stroke Association leaflet?","volume":"17","author":"Suleman","year":"2025","journal-title":"Cureus."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"e55939","DOI":"10.2196\/55939","article-title":"Evaluating the efficacy of Chatgpt as a patient education tool in prostate Cancer: multimetric assessment","volume":"26","author":"Gibson","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref29","doi-asserted-by":"publisher","first-page":"2070","DOI":"10.1111\/j.1572-0241.2007.01325.x","article-title":"A systematic review of patient inflammatory bowel disease information resources on the world wide web","volume":"102","author":"Bernard","year":"2007","journal-title":"Am J Gastroenterol"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.3390\/publications13020023","article-title":"Evaluation of content quality of online health information by global quality score: a case study of researchers misnaming it and citing secondary sources","volume":"13","author":"Yeung","year":"2025","journal-title":"Publications"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1136\/jech.53.2.105","article-title":"An instrument for judging the quality of written consumer health information on treatment choices","volume":"53","author":"Charnock","year":"1999","journal-title":"J Epidemiol Community Health"},{"key":"ref32","doi-asserted-by":"publisher","first-page":"e70445","DOI":"10.1002\/jeo2.70445","article-title":"Refining patient education on autologous chondrocyte implantation for chondral lesions of the knee: a fine-tuned Chatgpt-4o model improves readability and quality","volume":"12","author":"Alsadaan","year":"2025","journal-title":"J Exp Orthop"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1186\/s13054-025-05302-0","article-title":"Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study","volume":"29","author":"Workum","year":"2025","journal-title":"Crit Care"},{"key":"ref34","doi-asserted-by":"publisher","first-page":"6143","DOI":"10.1109\/jbhi.2024.3483816","article-title":"Benchmarking large language models in evidence-based medicine","volume":"29","author":"Li","year":"2025","journal-title":"IEEE J Biomed Health Inform"},{"key":"ref35","doi-asserted-by":"publisher","first-page":"e70047","DOI":"10.2196\/70047","article-title":"Evaluating large language models for preoperative patient education in superior capsular reconstruction: comparative study of Claude, Gpt, and Gemini","volume":"8","author":"Liu","year":"2025","journal-title":"JMIR Perioper Med"},{"key":"ref36","doi-asserted-by":"publisher","first-page":"1701344","DOI":"10.3389\/fmed.2025.1701344","article-title":"Evaluating the quality of large language model-generated preoperative patient education material: a comparative study across models and surgery types","volume":"12","author":"Ma","year":"2025","journal-title":"Front Med"},{"key":"ref37","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for Llms","volume":"7","author":"Wang","year":"2024","journal-title":"NPJ Digit Med"},{"key":"ref38","doi-asserted-by":"publisher","first-page":"14045","DOI":"10.1038\/s41598-023-41032-5","article-title":"Fabrication and errors in the bibliographic citations generated by Chatgpt","volume":"13","author":"Walters","year":"2023","journal-title":"Sci Rep"},{"key":"ref39","doi-asserted-by":"publisher","first-page":"103275","DOI":"10.1016\/j.msksp.2025.103275","article-title":"Chatgpt is a comprehensive education tool for patients with patellar tendinopathy, but it currently lacks accuracy and readability","volume":"76","author":"Deng","year":"2025","journal-title":"Musculoskelet Sci Pract"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"1801829","DOI":"10.3389\/fpubh.2026.1801829","article-title":"Challenges of using generative Ai for patient education in chronic heart failure: an evaluation of content quality, readability, and Actionability in cross-platform Llm-generated texts","volume":"14","author":"Wang","year":"2026","journal-title":"Front Public Health"}],"container-title":["Frontiers in Public Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpubh.2026.1900281\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T15:37:08Z","timestamp":1787153828000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpubh.2026.1900281\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,19]]},"references-count":40,"alternative-id":["10.3389\/fpubh.2026.1900281"],"URL":"https:\/\/doi.org\/10.3389\/fpubh.2026.1900281","relation":{},"ISSN":["2296-2565"],"issn-type":[{"value":"2296-2565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,19]]},"article-number":"1900281"}}