{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T18:36:59Z","timestamp":1791484619840,"version":"4.3.4"},"reference-count":58,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Scars and keloids impose significant physical and psychological burdens on patients, often leading to functional limitations, cosmetic concerns, and mental health issues such as anxiety or depression. Patients increasingly turn to online platforms for information; however, existing web-based resources on scars and keloids are frequently unreliable, fragmented, or difficult to understand. Large language models such as GPT-4 show promise for delivering medical information, but their accuracy, readability, and potential to generate hallucinated content require validation for patient education applications.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to systematically evaluate GPT-4\u2019s performance in providing patient education on scars and keloids, focusing on its accuracy, reliability, readability, and reference quality.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This study involved collecting 354 questions from Reddit communities (r\/Keloids, r\/SCAR, and r\/PlasticSurgery), covering topics including treatment options, pre- and postoperative care, and psychological impacts. Each question was input into GPT-4 in independent sessions to mimic real-world patient interactions. Responses were evaluated using multiple tools: the Patient Education Materials Assessment Tool-Artificial Intelligence for understandability and actionability, DISCERN-AI for treatment information quality, the Global Quality Scale for overall information quality, and standard readability metrics (Flesch Reading Ease score, and Gunning Fog Index). Three plastic surgeons used the Natural Language Assessment Tool for Artificial Intelligence to rate the accuracy, safety, and clinical appropriateness, while the Reference Evaluation for Artificial Intelligence tool validated references for reference hallucination, relevance, and source quality. We conducted the same analysis to assess the quality of GPT-4\u2013generated content in response to questions from 3 medical websites.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n GPT-4 demonstrated high accuracy and reliability. The Patient Education Materials Assessment Tool-Artificial Intelligence showed 75.5% understandability, DISCERN-AI rated responses as \u201cgood\u201d (26.3\/35), and the Global Quality Scale score was 4.28 of 5. Surgeons\u2019 evaluations averaged 3.94 to 4.43 out of 5 across dimensions (accuracy 3.9, SD 0.7; safety 4.3, SD 0.8; clinical appropriateness 4.4, SD 0.5; actionability 4.1, SD 0.8; and effectiveness 4.1, SD 0.8). Readability analyses indicated moderate complexity (Flesch Reading Ease Score: 50.13; Gunning Fog Index: 12.68), corresponding to a 12th-grade reading level. Reference Evaluation for Artificial Intelligence identified 11.8% (383\/3250) hallucinated references, while 88.2% (2867\/3250) of references were real, with 95.1% (2724\/2867) from authoritative sources (eg, government guidelines and the literature). The overall results about questions from medical websites were consistent with the answers to Reddit questions.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n GPT-4 has serious potential as a patient education tool for scars and keloids, offering reliable and accurate information. However, improvements in readability (to align with sixth to eighth grade standards) and reduction of reference hallucinations are essential to enhance accessibility and trustworthiness. Future large language model optimizations should prioritize simplifying medical language and strengthening reference validation mechanisms to maximize clinical utility.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/78838","type":"journal-article","created":{"date-parts":[[2025,12,30]],"date-time":"2025-12-30T02:15:09Z","timestamp":1767060909000},"page":"e78838-e78838","source":"Crossref","is-referenced-by-count":2,"title":["Evaluating GPT-4 Responses on Scars or Keloids for Patient Education: Large Language Model Evaluation Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-5254-8889","authenticated-orcid":false,"given":"Mingjun","family":"Rao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3106-5253","authenticated-orcid":false,"given":"Tang","family":"Xiujun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5790-9087","authenticated-orcid":false,"given":"Wang","family":"Haoyu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,2,27]]},"reference":[{"issue":"11","key":"R1","doi-asserted-by":"publisher","DOI":"10.3390\/nano10112179","article-title":"Scar-free healing: current concepts and future perspectives","volume":"10","author":"Stoica","journal-title":"Nanomaterials (Basel)"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1111\/wrr.12637","article-title":"Risk factors for hypertrophic burn scar pain, pruritus, and paresthesia development","volume":"26","author":"Xiao","journal-title":"Wound Repair Regen"},{"issue":"4","key":"R3","doi-asserted-by":"publisher","DOI":"10.1097\/GOX.0000000000003522","article-title":"Daily quality-of-life impact of scars: an interview-based foundational study of patient-reported themes","volume":"9","author":"Hsieh","journal-title":"Plast Reconstr Surg Glob Open"},{"issue":"4","key":"R4","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1093\/jbcr\/irad016","article-title":"Considerations in determining the severity of burn scar contractures with focus on the knee joint","volume":"44","author":"Schouten","journal-title":"J Burn Care Res"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1097\/SPV.0b013e3181cc8702","article-title":"Outcomes of revision perineoplasty for persistent postpartum dyspareunia","volume":"16","author":"Woodward","journal-title":"Female Pelvic Med Reconstr Surg"},{"issue":"4","key":"R6","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1016\/j.fsc.2023.06.005","article-title":"Scar therapy of skin","volume":"31","author":"Waibel","journal-title":"Facial Plast Surg Clin North Am"},{"issue":"9","key":"R7","doi-asserted-by":"publisher","first-page":"2739","DOI":"10.1111\/jocd.13945","article-title":"Scar tissue I wish you saw: patient expectations regarding scar treatment","volume":"20","author":"Gomolin","journal-title":"J Cosmet Dermatol"},{"issue":"4","key":"R8","doi-asserted-by":"publisher","first-page":"224","DOI":"10.3109\/14764170903341723","article-title":"Scar characteristics and treatment expectations: a survey of 589 patients","volume":"11","author":"Cho","journal-title":"J Cosmet Laser Ther"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1016\/j.burns.2017.09.003","article-title":"Below the surface: parents\u2019 views on the factors that influence treatment adherence in paediatric burn scar management - a qualitative study","volume":"44","author":"Andrews","journal-title":"Burns"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.01.007","article-title":"Health literacy and the internet: an exploratory study on the 2013 HINTS survey","volume":"58","author":"Jiang","journal-title":"Comput Human Behav"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1177\/2055207620948996","article-title":"Factors affecting the quality and reliability of online health information","volume":"6","author":"Battineni","journal-title":"Digit Health"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.pec.2019.08.008","article-title":"Health information overload among health consumers: a scoping review","volume":"103","author":"Khaleel","journal-title":"Patient Educ Couns"},{"issue":"4","key":"R13","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1097\/01.BCR.0000075844.04297.D9","article-title":"An evaluation of internet sites for burn scar management","volume":"24","author":"Bohacek","journal-title":"J Burn Care Rehabil"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2024.108013","article-title":"ChatGPT in healthcare: a taxonomy and systematic review","volume":"245","author":"Li","journal-title":"Comput Methods Programs Biomed"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"2618","DOI":"10.3390\/ai5040126","article-title":"ChatGPT: transforming healthcare with AI","volume":"5","author":"Neha","journal-title":"AI"},{"issue":"12","key":"R16","doi-asserted-by":"publisher","first-page":"971","DOI":"10.1016\/j.molmed.2023.08.012","article-title":"Medical education empowered by generative artificial intelligence large language models","volume":"29","author":"Jowsey","journal-title":"Trends Mol Med"},{"issue":"6","key":"R17","doi-asserted-by":"publisher","first-page":"1341","DOI":"10.1093\/jamia\/ocae067","article-title":"Can large language models provide secondary reliable opinion on treatment options for dermatological diseases?","volume":"31","author":"Iqbal","journal-title":"J Am Med Inform Assoc"},{"key":"R18","unstructured":"Lee R . Close encounters of the AI kind: the increasingly human-like way people are engaging with language models. Elon University; URL: https:\/\/imaginingthedigitalfuture.org\/reports-and-publications\/close-encounters-of-the-ai-kind\/close-encounters-of-the-ai-kind-main-report [Accessed 26-11-2025]"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.2147\/JMDH.S502351","article-title":"Large language models in healthcare: a bibliometric analysis and examination of research trends","volume":"18","author":"Gencer","journal-title":"J Multidiscip Healthc"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2023.102666","article-title":"Empowering elderly care with intelligent IoT-driven smart toilets for home-based infectious health monitoring","volume":"144","author":"Kumar","journal-title":"Artif Intell Med"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Kumar D Rawat KS Sood SK . Revolution of artificial intelligence and IOT in healthcare: a keyword co-occurrence network analysis using CiteSpace. In: Shukla AK Thakur DG Arabkoohsar A , editors. Recent Advances in Mechanical Engineering. 2023:231-237. [doi: 10.1007\/978-981-99-2349-6_20] ISBN: 9789819923496","DOI":"10.1007\/978-981-99-2349-6_20"},{"issue":"2","key":"R22","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1111\/wrr.13160","article-title":"An analysis of keloid patient questions on Reddit","volume":"32","author":"Wang","journal-title":"Wound Repair Regen"},{"issue":"3","key":"R23","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1089\/thy.2023.0491","article-title":"Evaluating ChatGPT responses on thyroid nodules for patient education","volume":"34","author":"Campbell","journal-title":"Thyroid"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/55939","article-title":"Evaluating the efficacy of ChatGPT as a patient education tool in prostate cancer: multimetric assessment","volume":"26","author":"Gibson","journal-title":"J Med Internet Res"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","first-page":"325","DOI":"10.2106\/JBJS.N.01189","article-title":"Orthopaedic patient information on the world wide web: an essential review","volume":"98","author":"Cassidy","journal-title":"J Bone Joint Surg Am"},{"key":"R26","unstructured":"Readable. Our Readability Checker helps you to communicate clearly. URL: https:\/\/readable.com [Accessed 24-02-2026]"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"Alkaissi","journal-title":"Cureus"},{"issue":"3","key":"R28","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1111\/j.1365-4632.2010.04705.x","article-title":"Internet use by dermatology outpatients to search for health information","volume":"50","author":"AlGhamdi","journal-title":"Int J Dermatol"},{"issue":"4","key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/36463","article-title":"Consumers\u2019 evaluation of web-based health information quality: meta-analysis","volume":"24","author":"Zhang","journal-title":"J Med Internet Res"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1007\/s10639-024-13065-4","article-title":"Exploring learners\u2019 experiences and perceptions of ChatGPT as a learning tool in higher education","volume":"30","author":"Monib","journal-title":"Educ Inf Technol"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1186\/s12879-023-08724-9","article-title":"ChatGPT and mycosis- a new weapon in the knowledge battlefield","volume":"23","author":"Jin","journal-title":"BMC Infect Dis"},{"issue":"11","key":"R32","doi-asserted-by":"publisher","first-page":"1923","DOI":"10.1177\/00031348251341956","article-title":"Can American Board of Surgery in Training Examinations be passed by large language models? Comparative assessment of Gemini, Copilot, and ChatGPT","volume":"91","author":"Sanli","journal-title":"Am Surg"},{"issue":"9","key":"R33","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.69612","article-title":"Comparison of Gemini Advanced and ChatGPT 4.0\u2019s performances on the Ophthalmology Resident Ophthalmic Knowledge Assessment Program (OKAP) examination review question banks","volume":"16","author":"Gill","journal-title":"Cureus"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1097\/us9.0000000000000089","article-title":"Evaluating the performance of artificial intelligence chatbots in answering urology questions derived from guidelines or board examinations: a systematic review","author":"Tan","journal-title":"Urol Sci"},{"issue":"20","key":"R35","doi-asserted-by":"publisher","DOI":"10.3390\/jcm12206524","article-title":"Using generative artificial intelligence tools in cosmetic surgery: a study on rhinoplasty, facelifts, and blepharoplasty procedures","volume":"12","author":"Lim","journal-title":"J Clin Med"},{"issue":"17","key":"R36","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12171781","article-title":"Online patient education in obstructive sleep apnea: ChatGPT versus Google search","volume":"12","author":"Incerti Parenti","journal-title":"Healthcare (Basel)"},{"issue":"11","key":"R37","doi-asserted-by":"publisher","DOI":"10.14309\/ctg.0000000000000765","article-title":"Digesting digital health: a study of appropriateness and readability of ChatGPT-generated gastroenterological information","volume":"15","author":"Toiv","journal-title":"Clin Transl Gastroenterol"},{"key":"R38","unstructured":"Veronique I . Report on the condition of education 2024. Institute of Education Sciences; 2024. URL: https:\/\/nces.ed.gov\/pubs2024\/2024144.pdf"},{"issue":"2","key":"R39","doi-asserted-by":"publisher","first-page":"835","DOI":"10.15562\/bmj.v10i2.2619","article-title":"The relationship between skin color and keloid","volume":"10","author":"Oei","journal-title":"Bali Med J"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-03035-2","article-title":"Comparison of the readability of ChatGPT and Bard in medical communication: a meta-analysis","volume":"25","author":"DeTemple","journal-title":"BMC Med Inform Decis Mak"},{"issue":"4","key":"R41","doi-asserted-by":"publisher","DOI":"10.1177\/14604582251388879","article-title":"Quality and readability of chatbot responses to patient questions: a systematic cross-sectional meta-synthesis","volume":"31","author":"Whittaker","journal-title":"Health Informatics J"},{"key":"R42","doi-asserted-by":"crossref","unstructured":"Li M Zhang Y Li Z et al. From quantity to quality: boosting LLM performance with self-guided data selection for instruction tuning. Presented at: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics; Jun 16-21, 2024:7602-7635; Mexico City, Mexico. [doi: 10.18653\/v1\/2024.naacl-long.421]","DOI":"10.18653\/v1\/2024.naacl-long.421"},{"issue":"4","key":"R43","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000489","article-title":"Biomedical text readability after hypernym substitution with fine-tuned large language models","volume":"3","author":"Swanson","journal-title":"PLOS Digit Health"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1016\/j.wneu.2025.124423","article-title":"Prompt engineering and follow-up questioning improves the readability of spine surgery questions in large language models","volume":"203","author":"Daulat","journal-title":"World Neurosurg"},{"key":"R45","doi-asserted-by":"crossref","unstructured":"Aljohani M Hou J Kommu S Wang X . A comprehensive survey on the trustworthiness of large language models in healthcare. Presented at: Proceedings of the 2025 Findings of the Association for Computational Linguistics EMNLP\u201925; Nov 4-9, 2025:6720-6748; Suzhou, China. [doi: 10.18653\/v1\/2025.findings-emnlp.356]","DOI":"10.18653\/v1\/2025.findings-emnlp.356"},{"key":"R46","doi-asserted-by":"crossref","unstructured":"Kim Y Jeong H Chen S et al. Medical hallucination in foundation models and their impact on healthcare. Health Systems and Quality Improvement. Preprint posted online on 2025. [doi: 10.1101\/2025.02.28.25323115]","DOI":"10.1101\/2025.02.28.25323115"},{"issue":"5","key":"R47","doi-asserted-by":"publisher","first-page":"856","DOI":"10.3390\/math13050856","article-title":"Hallucination mitigation for retrieval-augmented large language models: a review","volume":"13","author":"Zhang","journal-title":"Mathematics"},{"issue":"1","key":"R48","doi-asserted-by":"publisher","first-page":"62","DOI":"10.69554\/NXXB8234","article-title":"Strategies to mitigate hallucinations in large language models","volume":"10","author":"Bhattacharya","journal-title":"AMA"},{"issue":"21","key":"R49","doi-asserted-by":"publisher","first-page":"4227","DOI":"10.3390\/electronics14214227","article-title":"Evaluating retrieval-augmented generation variants for clinical decision support: hallucination mitigation and secure on-premises deployment","volume":"14","author":"Wo\u0142k","journal-title":"Electronics (Basel)"},{"key":"R50","doi-asserted-by":"crossref","unstructured":"Agrawal G Kumarage T Alghamdi Z Liu H . Can knowledge graphs reduce hallucinations in LLMs?: a survey. Presented at: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics; Jun 16-21, 2024:3947-3960; Mexico City, Mexico. [doi: 10.18653\/v1\/2024.naacl-long.219]","DOI":"10.18653\/v1\/2024.naacl-long.219"},{"issue":"1","key":"R51","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-93601-5","article-title":"Mitigating spatial hallucination in large language models for path planning via prompt engineering","volume":"15","author":"Zhang","journal-title":"Sci Rep"},{"issue":"1","key":"R52","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-024-00717-2","article-title":"Current applications and challenges in large language models for patient care: a systematic review","volume":"5","author":"Busch","journal-title":"Commun Med (Lond)"},{"key":"R53","unstructured":"Ferrag MA Tihanyi N Debbah M . From LLM reasoning to autonomous ai agents: a comprehensive review. arXiv. Preprint posted online on Apr 28, 2025. [doi: 10.48550\/arXiv.2504.19678]"},{"key":"R54","doi-asserted-by":"publisher","DOI":"10.2196\/59505","article-title":"Multimodal large language models in health care: applications, challenges, and future outlook","volume":"26","author":"AlSaad","journal-title":"J Med Internet Res"},{"key":"R55","unstructured":"Reddit user age, gender, & demographics. Exploding Topics. 2025. URL: https:\/\/explodingtopics.com\/blog\/reddit-users [Accessed 26-11-2025]"},{"key":"R56","unstructured":"Calloway C . Why do different LLMs give different answers to the same question? Model uncertainty and variability in LLM-based intrusion detection systems ranking. Norfolk State University; 2025. URL: https:\/\/digitalcommons.odu.edu\/cgi\/viewcontent.cgi?article=1123&context=covacci-undergraduateresearch [Accessed 20-05-2024]"},{"key":"R57","doi-asserted-by":"crossref","unstructured":"Yona G Aharoni R Geva M . Can large language models faithfully express their intrinsic uncertainty in words? Presented at: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing; Nov 12-16, 2024:632-702; Miami, FL. [doi: 10.18653\/v1\/2024.emnlp-main.443]","DOI":"10.18653\/v1\/2024.emnlp-main.443"},{"key":"R58","doi-asserted-by":"crossref","unstructured":"Fu T Conde J Mart\u00ednez G Grandury M Reviriego P . Multiple choice questions: reasoning makes large language models (LLMS) more self-confident even when they are wrong. arXiv. Preprint posted online on Jan 16, 2025. [doi: 10.48550\/arXiv.2501.09775]","DOI":"10.1109\/MIS.2026.3660833"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e78838","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:11:37Z","timestamp":1782843097000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e78838"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,27]]},"references-count":58,"URL":"https:\/\/doi.org\/10.2196\/78838","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,27]]},"article-number":"v14i11e78838"}}