{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T16:40:24Z","timestamp":1791996024390,"version":"4.4.0"},"reference-count":44,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Cancer"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) such as ChatGPT (OpenAI) are increasingly discussed as potential tools for patient education in health care. In radiation oncology, where patients are often confronted with complex medical terminology and complex treatment plans, LLMs may support patient understanding and promote more active participation in care. However, the readability, accuracy, completeness, and overall acceptance of LLM-generated medical content remain underexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the potential of ChatGPT-4 as a supplementary tool for patient education in the context of lung cancer radiotherapy by assessing the readability, content quality, and perceived usefulness of artificial intelligence\u2013generated responses from both clinician and patient perspectives.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A total of 8 frequently asked questions about radiotherapy for lung cancer were developed based on clinical experience from a team of clinicians specialized in lung cancer treatment at a university hospital. The questions were submitted individually to ChatGPT-4o (version as of July 2024) using the prompt: \u201cI am a lung cancer patient looking for answers to the following questions.\u201d Responses were evaluated using three approaches: (1) a readability analysis applying the Modified Flesch Reading Ease (FRE) formula for German and the 4th Vienna Formula (WSTF); (2) a multicenter expert evaluation by 6 multidisciplinary clinicians (radiation oncologists, medical oncologists, and thoracic surgeons) specialized in lung cancer treatment using a 5-point Likert scale to assess relevance, correctness, and completeness; and (3) a patient evaluation during the first follow-up appointment after radiotherapy, assessing comprehensibility, accuracy, relevance, trustworthiness, and willingness to use ChatGPT for future medical questions.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Readability analysis classified most responses as \u201cvery difficult to read\u201d (university level) or \u201cdifficult to read\u201d (upper secondary school), likely due to the use of medical language and long sentence structures. Clinician assessments yielded high scores for relevance (mean 4.5, SD 0.52) and correctness (mean 4.3, SD 0.65), but completeness received slightly lower ratings (mean 3.9, SD 0.59). A total of 30 patients rated the responses positively for clarity (mean 4.4, SD 0.61) and relevance (mean 4.3, SD 0.64), but lower for trustworthiness (mean 3.8, SD 0.68) and usability (mean 3.7, SD 0.73). No harmful misinformation was identified in the responses.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n ChatGPT-4 shows promise as a supplementary tool for patient education in radiation oncology. While patients and clinicians appreciated the clarity and relevance of the information, limitations in completeness, trust, and readability highlight the need for clinician oversight and further optimization of LLM-generated content. Future developments should focus on improving accessibility, integrating real-time readability adaptation, and establishing standardized evaluation frameworks to ensure safe and effective clinical use.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/69783","type":"journal-article","created":{"date-parts":[[2025,4,29]],"date-time":"2025-04-29T16:55:08Z","timestamp":1745945708000},"page":"e69783-e69783","source":"Crossref","is-referenced-by-count":8,"title":["Assessing ChatGPT\u2019s Educational Potential in Lung Cancer Radiotherapy From Clinician and Patient Perspectives: Content Quality and Readability Analysis"],"prefix":"10.2196","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-3567-5628","authenticated-orcid":false,"given":"Cedric","family":"Richlitzki","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-6224-1910","authenticated-orcid":false,"given":"Sina","family":"Mansoorian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6260-5249","authenticated-orcid":false,"given":"Lukas","family":"K\u00e4smann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1288-809X","authenticated-orcid":false,"given":"Mircea Gabriel","family":"Stoleriu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1952-399X","authenticated-orcid":false,"given":"Julia","family":"Kovacs","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9579-8102","authenticated-orcid":false,"given":"Wulf","family":"Sienel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1471-3290","authenticated-orcid":false,"given":"Diego","family":"Kauffmann-Guerrero","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1039-2090","authenticated-orcid":false,"given":"Thomas","family":"Duell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2091-3603","authenticated-orcid":false,"given":"Nina Sophie","family":"Schmidt-Hegemann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1287-7825","authenticated-orcid":false,"given":"Claus","family":"Belka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8709-7252","authenticated-orcid":false,"given":"Stefanie","family":"Corradini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3779-1398","authenticated-orcid":false,"given":"Chukwuka","family":"Eze","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,8,13]]},"reference":[{"issue":"13","key":"R1","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","journal-title":"N Engl J Med"},{"issue":"10","key":"R2","doi-asserted-by":"publisher","first-page":"1437","DOI":"10.1001\/jamaoncol.2023.2947","article-title":"Assessment of artificial intelligence chatbot responses to top searched queries about cancer","volume":"9","author":"Pan","journal-title":"JAMA Oncol"},{"issue":"16","key":"R3","doi-asserted-by":"publisher","first-page":"2229","DOI":"10.1055\/a-2302-8604","article-title":"Identifying ChatGPT-written patient education materials using text analysis and readability","volume":"41","author":"Monje","journal-title":"Am J Perinatol"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/46900","article-title":"Appropriateness and comprehensiveness of using ChatGPT for perioperative patient education in thoracic surgery in different language contexts: survey study","volume":"12","author":"Shao","journal-title":"Interact J Med Res"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"8","key":"R6","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1002\/acm2.12942","article-title":"A review of patient questions from physicist-patient consults","volume":"21","author":"Atwood","journal-title":"J Appl Clin Med Phys"},{"issue":"5","key":"R7","doi-asserted-by":"publisher","first-page":"923","DOI":"10.1093\/jamia\/ocaa229","article-title":"Physicians\u2019 electronic inbox work patterns and factors associated with high inbox work duration","volume":"28","author":"Akbar","journal-title":"J Am Med Inform Assoc"},{"issue":"9","key":"R8","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1080\/00016489.2023.2254809","article-title":"Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology department","volume":"143","author":"Nielsen","journal-title":"Acta Otolaryngol"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"721","DOI":"10.3350\/cmh.2023.0089","article-title":"Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma","volume":"29","author":"Yeo","journal-title":"Clin Mol Hepatol"},{"issue":"10","key":"R10","doi-asserted-by":"publisher","first-page":"1459","DOI":"10.1001\/jamaoncol.2023.2954","article-title":"Use of artificial intelligence chatbots for cancer treatment information","volume":"9","author":"Chen","journal-title":"JAMA Oncol"},{"issue":"7947","key":"R11","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1038\/d41586-023-00288-7","article-title":"ChatGPT: five priorities for research","volume":"614","author":"van Dis","journal-title":"Nature New Biol"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3389\/fradi.2023.1224682","article-title":"Surviving ChatGPT in healthcare","volume":"3","author":"Liu","journal-title":"Front Radiol"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.urology.2023.07.010","article-title":"Caution! AI bot has entered the patient chat: ChatGPT has limitations in providing accurate urologic healthcare advice","volume":"180","author":"Whiles","journal-title":"Urology"},{"issue":"9","key":"R14","doi-asserted-by":"publisher","first-page":"624","DOI":"10.1038\/s41571-023-00798-3","article-title":"The global burden of lung cancer: current status and future trends","volume":"20","author":"Leiter","journal-title":"Nat Rev Clin Oncol"},{"issue":"12","key":"R15","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare9121740","article-title":"Online health information seeking behavior: a systematic review","volume":"9","author":"Jia","journal-title":"Healthcare (Basel)"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick","volume":"32","author":"FLESCH","journal-title":"J Appl Psychol"},{"key":"R17","unstructured":"Amstad T . Wie Verst\u00e4ndlich Sind Unsere Zeitungen [Book in German]. Studenten-Schreib-Service; 1978. URL: https:\/\/books.google.de\/books?id=kiI7vwEACAAJ [Accessed 18-08-2024]"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Bamberger R Vanecek E . Lesen-Verstehen-Lernen-Schreiben: Die Schwierigkeitsstufen von Texten in Deutscher Sprache [Book in German]. Jugend und Volk; 1984. URL: https:\/\/books.google.de\/books?id=TElTAAAACAAJ [Accessed 18-08-2024]","DOI":"10.2307\/3530491"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1007\/s13187-021-02095-y","article-title":"Understanding the pathway of cancer information seeking: cancer information services as a supplement to information from other sources","volume":"38","author":"Reifegerste","journal-title":"J Cancer Educ"},{"issue":"3","key":"R20","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1016\/j.ijrobp.2016.06.2449","article-title":"Radiation oncology and online patient education materials: deviating from NIH and AMA recommendations","volume":"96","author":"Prabhu","journal-title":"Int J Radiat Oncol Biol Phys"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1016\/j.prro.2016.07.008","article-title":"Online patient information from radiation oncology departments is too complex for the general population","volume":"7","author":"Rosenberg","journal-title":"Pract Radiat Oncol"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Wang L Wan Z Ni C et al. A systematic review of ChatGPT and other conversational large language models in healthcare. medRxiv. Apr 27, 2024:2024.04.26.24306390. [doi: 10.1101\/2024.04.26.24306390] [Medline: 38712148]","DOI":"10.1101\/2024.04.26.24306390"},{"issue":"6","key":"R23","doi-asserted-by":"publisher","first-page":"313","DOI":"10.4274\/tjo.galenos.2024.28234","article-title":"Readability and appropriateness of responses generated by ChatGPT 3.5, ChatGPT 4.0, Gemini, and Microsoft Copilot for FAQs in refractive surgery","volume":"54","author":"Ayd\u0131n","journal-title":"Turk J Ophthalmol"},{"issue":"14","key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2024.e34391","article-title":"Benchmarking four large language models\u2019 performance of addressing Chinese patients\u2019 inquiries about dry eye disease: a two-phase study","volume":"10","author":"Shi","journal-title":"Heliyon"},{"issue":"6","key":"R25","doi-asserted-by":"publisher","first-page":"1751","DOI":"10.1002\/ohn.927","article-title":"Enhancing health literacy: evaluating the readability of patient handouts revised by ChatGPT\u2019s large language model","volume":"171","author":"Swisher","journal-title":"Otolaryngol Head Neck Surg"},{"issue":"5","key":"R26","doi-asserted-by":"publisher","first-page":"2522","DOI":"10.1007\/s00464-024-10720-2","article-title":"Large language models and bariatric surgery patient education: a comparative readability analysis of GPT-3.5, GPT-4, Bard, and online institutional resources","volume":"38","author":"Srinivasan","journal-title":"Surg Endosc"},{"issue":"2 D","key":"R27","doi-asserted-by":"publisher","DOI":"10.6004\/jnccn.2023.7334","article-title":"Enhancing readability of online patient-facing content: the role of AI chatbots in improving cancer information accessibility","volume":"22","author":"Abreu","journal-title":"J Natl Compr Canc Netw"},{"issue":"4","key":"R28","doi-asserted-by":"publisher","DOI":"10.1148\/ryct.230115","article-title":"Evaluating ChatGPT\u2019s accuracy in lung cancer prevention and screening recommendations","volume":"5","author":"Haver","journal-title":"Radiol Cardiothorac Imaging"},{"issue":"5","key":"R29","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230922","article-title":"How AI responds to common lung cancer questions: ChatGPT vs Google Bard","volume":"307","author":"Rahsepar","journal-title":"Radiology"},{"issue":"3","key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/j.adro.2023.101400","article-title":"Exploring capabilities of large language models such as ChatGPT in radiation oncology","volume":"9","author":"Dennst\u00e4dt","journal-title":"Adv Radiat Oncol"},{"issue":"3","key":"R31","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1016\/j.canrad.2023.11.005","article-title":"Evaluating ChatGPT to test its robustness as an interactive information database of radiation oncology and to assess its responses to common queries from radiotherapy patients: a single institution investigation","volume":"28","author":"Pandey","journal-title":"Cancer Radiother"},{"issue":"4","key":"R32","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.4630","article-title":"Quality of large language model responses to radiation oncology patient care questions","volume":"7","author":"Yalamanchili","journal-title":"JAMA Netw Open"},{"issue":"8","key":"R33","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.44316","article-title":"ChatGPT in medical education and research: a boon or a bane?","volume":"15","author":"Jeyaraman","journal-title":"Cureus"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.3389\/fsurg.2022.862322","article-title":"Legal and ethical consideration in artificial intelligence in healthcare: who takes responsibility?","volume":"9","author":"Naik","journal-title":"Front Surg"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.2196\/48009","article-title":"Ethical considerations of using ChatGPT in health care","volume":"25","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"6","key":"R36","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","journal-title":"JAMA Intern Med"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/60807","article-title":"Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis","volume":"26","author":"Liu","journal-title":"J Med Internet Res"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-024-02579-z","article-title":"ChatGPT has a language problem \u2014 but science can fix it","volume":"doi","author":"Petri\u0107 Howe","journal-title":"Nature New Biol"},{"issue":"3","key":"R39","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.55991","article-title":"Comparing the performance of popular large language models on the National Board of Medical Examiners sample questions","volume":"16","author":"Abbas","journal-title":"Cureus"},{"issue":"8","key":"R40","doi-asserted-by":"publisher","first-page":"1245","DOI":"10.1007\/s00234-024-03371-6","article-title":"Can large language models pass official high-grade exams of the European Society of Neuroradiology courses? A direct comparison between OpenAI chatGPT 3.5, OpenAI GPT4 and Google Bard","volume":"66","author":"D\u2019Anna","journal-title":"Neuroradiology"},{"issue":"3","key":"R41","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1007\/s00066-024-02342-3","article-title":"Patient- and clinician-based evaluation of large language models for patient education in prostate cancer radiotherapy","volume":"201","author":"Trapp","journal-title":"Strahlenther Onkol"},{"issue":"5","key":"R42","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1093\/oncolo\/oyae009","article-title":"Comparison of large language models in answering immuno-oncology questions: a cross-sectional study","volume":"29","author":"Iannantuono","journal-title":"Oncologist"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1477898","article-title":"Large language models in patient education: a scoping review of applications in medicine","volume":"11","author":"Aydin","journal-title":"Front Med (Lausanne)"},{"issue":"5","key":"R44","doi-asserted-by":"publisher","DOI":"10.1056\/aioa2300151","article-title":"Comparative evaluation of LLMs in clinical oncology","volume":"1","author":"Rydzewski","journal-title":"NEJM AI"}],"container-title":["JMIR Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e69783","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:57:54Z","timestamp":1782842274000},"score":1,"resource":{"primary":{"URL":"https:\/\/cancer.jmir.org\/2025\/1\/e69783"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,13]]},"references-count":44,"URL":"https:\/\/doi.org\/10.2196\/69783","relation":{},"ISSN":["2369-1999"],"issn-type":[{"value":"2369-1999","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,13]]},"article-number":"v11i13e69783"}}