{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T05:54:58Z","timestamp":1790747698874,"version":"4.1.0"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T00:00:00Z","timestamp":1790726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Oral Health"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Recurrent aphthous stomatitis (RAS), commonly known as oral ulcers, is a frequent and painful oral mucosal condition that often prompts people to seek consumer health information before professional consultation. Although LLM-generated health advice has been studied in several clinical fields, multidimensional evidence focused specifically on RAS remains limited.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare the safety, accuracy, supportive communication\/empathy, information quality and transparency, and readability of five publicly accessible LLM-based chatbots when responding to researcher-developed consumer-oriented prompts about RAS.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Thirty researcher-developed, consumer-oriented English prompts covering seven RAS domains were submitted once, in separate sessions, to ChatGPT, Copilot, Gemini, Doubao, and DeepSeek, yielding 150 single-turn responses. Safety was the primary clinical outcome; accuracy, supportive communication\/empathy, and information-quality measures were secondary, and six formula-based readability indices were exploratory. Five reviewers scored each response independently before consensus adjudication. User-facing product labels, access conditions, prompts, reference checklists, complete responses, and statistical outputs are documented in the manuscript and supplementary materials.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Sixteen of 150 responses (10.7%) were consensus-coded as potentially unsafe or misleading. Safe-response rates ranged from 83.3% to 93.3%, with no detected between-chatbot difference (Cochran\u2019s Q\u2009=\u20092.957,\n                      <jats:italic>P<\/jats:italic>\n                      \u2009=\u20090.565). Potential safety concerns were observed in persistent or non-healing ulcers, cancer-related reassurance, home or traditional remedies, supplementation, pregnancy, and pediatric scenarios. No between-chatbot differences were detected for accuracy, supportive communication\/empathy, DISCERN, EQIP, JAMA, or GQS. Five of six readability formulas detected differences, but only 31 of 150 responses (20.7%) met the prespecified FKGL target of grade 6 or below.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>In this exploratory dataset, no between-chatbot differences were detected for safety, accuracy, supportive communication\/empathy, or information-quality scores, whereas formula-dependent readability differences were observed. Non-significant findings do not establish equivalence, and readability formulas do not establish comprehension. The results apply only to the recorded user-facing configurations and single responses to 30 researcher-developed English prompts during the defined access period; they do not demonstrate readiness for autonomous clinical or patient-facing deployment.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/froh.2026.1968034","type":"journal-article","created":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T04:56:20Z","timestamp":1790744180000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Safety, accuracy, information quality, readability, and supportive communication of LLM-based chatbots in responses to researcher-developed consumer-oriented prompts on recurrent aphthous stomatitis: an exploratory cross-sectional comparative study"],"prefix":"10.3389","volume":"7","author":[{"given":"Yulong","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Stomatology, The Second People\u2019s Hospital in Mudanjiang City","place":["Mudanjiang, Heilongjiang, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingjing","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Oral Medicine, Mudanjiang Stomatological Hospital","place":["Mudanjiang, Heilongjiang, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhaole","family":"Gong","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, The Second People\u2019s Hospital in Mudanjiang City","place":["Mudanjiang, Heilongjiang, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan","family":"Zhuang","sequence":"additional","affiliation":[{"name":"Department of Stomatology, The Affiliated Wuxi People\u2019s Hospital of Nanjing Medical University, Wuxi People\u2019s Hospital, Wuxi Medical Center, Nanjing Medical University","place":["Wuxi, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wu","family":"Dong","sequence":"additional","affiliation":[{"name":"Department of Stomatology, The Affiliated Wuxi People\u2019s Hospital of Nanjing Medical University, Wuxi People\u2019s Hospital, Wuxi Medical Center, Nanjing Medical University","place":["Wuxi, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,30]]},"reference":[{"key":"B1","author":"Plewa","year":"2026"},{"key":"B2","year":"2024"},{"key":"B3","year":"2024"},{"key":"B4","year":"2024"},{"key":"B5","volume-title":"Global Oral Health Status Report: Towards Universal Health Coverage for Oral Health by 2030","year":"2022"},{"key":"B6","volume-title":"Global Strategy on Digital Health 2020\u20132025","year":"2021"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1862","DOI":"10.1111\/odi.13704","article-title":"Oral potentially malignant disorders: a consensus report from an international seminar on nomenclature and classification","volume":"27","author":"Warnakulasuriya","year":"2021","journal-title":"Oral Dis"},{"key":"B8","volume-title":"Suspected Cancer: Recognition and Referral. NICE Guideline NG12","year":"2023"},{"key":"B9","volume-title":"Ethics and Governance of Artificial Intelligence for Health: WHO Guidance","year":"2021"},{"key":"B10","volume-title":"Ethics and Governance of Artificial Intelligence for Health: Guidance on Large Multi-Modal Models","year":"2024"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"B12","doi-asserted-by":"publisher","first-page":"210","DOI":"10.7326\/M23-2772","article-title":"Large language models in medicine: the potentials and pitfalls","volume":"177","author":"Omiye","year":"2024","journal-title":"Ann Intern Med"},{"key":"B13","doi-asserted-by":"publisher","first-page":"104620","DOI":"10.1016\/j.jbi.2024.104620","article-title":"Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis","volume":"151","author":"Wei","year":"2024","journal-title":"J Biomed Inform"},{"key":"B14","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"B15","doi-asserted-by":"publisher","first-page":"e2457879","DOI":"10.1001\/jamanetworkopen.2024.57879","article-title":"Large language models for chatbot health advice studies: a systematic review","volume":"8","author":"Huo","year":"2025","journal-title":"JAMA Netw Open"},{"key":"B16","doi-asserted-by":"publisher","first-page":"e001632","DOI":"10.1136\/bmjmed-2025-001632","article-title":"Reporting guideline for chatbot health advice studies: the chatbot assessment reporting tool (CHART) statement","volume":"4","year":"2025","journal-title":"BMJ Med"},{"key":"B17","doi-asserted-by":"publisher","first-page":"e083305","DOI":"10.1136\/bmj-2024-083305","article-title":"Reporting guidelines for chatbot health advice studies: explanation and elaboration for the chatbot assessment reporting tool (CHART)","volume":"390","year":"2025","journal-title":"Br Med J"},{"key":"B18","year":"2026"},{"key":"B19","year":"2026"},{"key":"B20","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1136\/jech.53.2.105","article-title":"DISCERN: an instrument for judging the quality of written consumer health information on treatment choices","volume":"53","author":"Charnock","year":"1999","journal-title":"J Epidemiol Community Health"},{"key":"B21","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1111\/j.1369-7625.2004.00273.x","article-title":"Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information","volume":"7","author":"Moult","year":"2004","journal-title":"Health Expect"},{"key":"B22","doi-asserted-by":"publisher","first-page":"2070","DOI":"10.1111\/j.1572-0241.2007.01325.x","article-title":"A systematic review of patient inflammatory bowel disease information resources on the world wide web","volume":"102","author":"Bernard","year":"2007","journal-title":"Am J Gastroenterol"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1244","DOI":"10.1001\/jama.1997.03540390074039","article-title":"Assessing, controlling, and assuring the quality of medical information on the internet: caveant lector et viewor\u2014let the reader and viewer beware","volume":"277","author":"Silberg","year":"1997","journal-title":"JAMA"},{"key":"B24","volume-title":"Automated Readability Index. AMRL-TR-66-220. Wright-Patterson Air Force Base","author":"Smith","year":"1967"},{"key":"B25","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1037\/h0076540","article-title":"A computer readability formula designed for machine scoring","volume":"60","author":"Coleman","year":"1975","journal-title":"J Appl Psychol"},{"key":"B26","volume-title":"Derivation of New Readability Formulas for Navy Enlisted Personnel. Research Branch Report 8-75","author":"Kincaid","year":"1975"},{"key":"B27","volume-title":"The Technique of Clear Writing","author":"Gunning","year":"1952"},{"key":"B28","first-page":"639","article-title":"SMOG grading-a new readability formula","volume":"12","author":"McLaughlin","year":"1969","journal-title":"J Reading"},{"key":"B29","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick","volume":"32","author":"Flesch","year":"1948","journal-title":"J Appl Psychol"},{"key":"B30","year":""},{"key":"B31","volume-title":"Health Literacy and Patient Safety: Help Patients Understand","author":"Weiss","year":"2007"},{"key":"B32","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychol Bull"},{"key":"B33","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"B34","author":"Kassambara","year":"2023"},{"key":"B35","volume-title":"R: A Language and Environment for Statistical Computing","year":"2024"},{"key":"B36","author":"Canty","year":"2024"},{"key":"B37","doi-asserted-by":"publisher","first-page":"338","DOI":"10.1111\/edt.13020","article-title":"Performance of artificial intelligence chatbots in responding to patient queries related to traumatic dental injuries: a comparative study","volume":"41","author":"Guven","year":"2025","journal-title":"Dent Traumatol"},{"key":"B38","doi-asserted-by":"publisher","first-page":"AIoa2300151","DOI":"10.1056\/AIoa2300151","article-title":"Comparative evaluation of LLMs in clinical oncology","volume":"1","author":"Rydzewski","year":"2024","journal-title":"NEJM AI"},{"key":"B39","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"key":"B40","doi-asserted-by":"publisher","first-page":"936","DOI":"10.1016\/j.surg.2023.12.014","article-title":"Performance of GPT-4 on surgical knowledge assessments","volume":"175","author":"Beaulieu-Jones","year":"2024","journal-title":"Surgery"},{"key":"B41","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1097\/UPJ.0000000000000490","article-title":"Comparison of ChatGPT and traditional patient education materials for men\u2019s health","volume":"11","author":"Shah","year":"2024","journal-title":"Urol Pract"},{"key":"B42","doi-asserted-by":"publisher","first-page":"1477898","DOI":"10.3389\/fmed.2024.1477898","article-title":"Large language models in patient education: a scoping review of applications in medicine","volume":"11","author":"Aydin","year":"2024","journal-title":"Front Med (Lausanne)"},{"key":"B43","doi-asserted-by":"publisher","first-page":"102938","DOI":"10.1016\/j.artmed.2024.102938","article-title":"MedExpQA: multilingual benchmarking of large language models for medical question answering","volume":"155","author":"Alonso","year":"2024","journal-title":"Artif Intell Med"},{"key":"B44","doi-asserted-by":"publisher","first-page":"e52784","DOI":"10.2196\/52784","article-title":"Influence of model evolution and system roles on ChatGPT\u2019s performance in Chinese medical licensing exams: comparative study","volume":"10","author":"Ming","year":"2024","journal-title":"JMIR Med Educ"},{"key":"B45","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesk\u00f3","year":"2023","journal-title":"NPJ Digit Med"},{"key":"B46","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"}],"container-title":["Frontiers in Oral Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/froh.2026.1968034\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T04:56:23Z","timestamp":1790744183000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/froh.2026.1968034\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,30]]},"references-count":46,"alternative-id":["10.3389\/froh.2026.1968034"],"URL":"https:\/\/doi.org\/10.3389\/froh.2026.1968034","relation":{},"ISSN":["2673-4842"],"issn-type":[{"value":"2673-4842","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,30]]},"article-number":"1968034"}}