{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T14:24:24Z","timestamp":1791469464168,"version":"4.3.4"},"reference-count":46,"publisher":"Harborside Press, LLC","issue":"2D","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p><jats:bold>Background:<\/jats:bold> Internet-based health education is increasingly vital in patient care. However, the readability of online information often exceeds the average reading level of the US population, limiting accessibility and comprehension. This study investigates the use of chatbot artificial intelligence to improve the readability of cancer-related patient-facing content. <jats:bold>Methods:<\/jats:bold> We used ChatGPT 4.0 to rewrite content about breast, colon, lung, prostate, and pancreas cancer across 34 websites associated with NCCN Member Institutions. Readability was analyzed using Fry Readability Score, Flesch-Kincaid Grade Level, Gunning Fog Index, and Simple Measure of Gobbledygook. The primary outcome was the mean readability score for the original and artificial intelligence (AI)\u2013generated content. As secondary outcomes, we assessed the accuracy, similarity, and quality using F1 scores, cosine similarity scores, and section 2 of the DISCERN instrument, respectively. <jats:bold>Results:<\/jats:bold> The mean readability level across the 34 websites was equivalent to a university freshman level (grade 13\u00b11.5). However, after ChatGPT\u2019s intervention, the AI-generated outputs had a mean readability score equivalent to a high school freshman education level (grade 9\u00b10.8). The overall F1 score for the rewritten content was 0.87, the precision score was 0.934, and the recall score was 0.814. Compared with their original counterparts, the AI-rewritten content had a cosine similarity score of 0.915 (95% CI, 0.908\u20130.922). The improved readability was attributed to simpler words and shorter sentences. The mean DISCERN score of the random sample of AI-generated content was equivalent to \u201cgood\u201d (28.5\u00b15), with no significant differences compared with their original counterparts. <jats:bold>Conclusions:<\/jats:bold> Our study demonstrates the potential of AI chatbots to improve the readability of patient-facing content while maintaining content quality. The decrease in requisite literacy after AI revision emphasizes the potential of this technology to reduce health care disparities caused by a mismatch between educational resources available to a patient and their health literacy.<\/jats:p>","DOI":"10.6004\/jnccn.2023.7334","type":"journal-article","created":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T16:52:33Z","timestamp":1715791953000},"source":"Crossref","is-referenced-by-count":54,"title":["Enhancing Readability of Online Patient-Facing Content: The Role of AI Chatbots in Improving Cancer Information Accessibility"],"prefix":"10.6004","volume":"22","author":[{"given":"Andres A.","family":"Abreu","sequence":"first","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gilbert Z.","family":"Murimwa","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emile","family":"Farah","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James W.","family":"Stewart","sequence":"additional","affiliation":[{"name":"Department of Surgery, Yale School of Medicine, New Haven, CT"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucia","family":"Zhang","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Rodriguez","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Sweetenham","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Herbert J.","family":"Zeh","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sam C.","family":"Wang","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricio M.","family":"Polanco","sequence":"additional","affiliation":[{"name":"Division of Surgical Oncology, Department of Surgery, UT Southwestern Medical Center, Dallas, TX"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4047","reference":[{"key":"B1","series-title":"JAMA","first-page":"2400","article-title":"Use of the internet and e-mail for health care information: results from a national survey","volume":"289","author":"Baker L","year":"2003","unstructured":"Baker L, Wagner TH, Singer S,  Use of the internet and e-mail for health care information: results from a national survey. JAMA 2003;289:2400\u20132406."},{"key":"B2","author":"Cohen RA","unstructured":"Cohen RA, Stussman B. Health information technology use among men and women aged 18\u201364: early release of estimates from the National Health Interview Survey, January\u2013June 2009. Accessed October 10, 2023. Available at: https:\/\/www.cdc.gov\/nchs\/data\/hestat\/healthinfo2009\/healthinfo2009.pdf"},{"key":"B3","series-title":"Support Care Cancer","first-page":"1189","article-title":"Internet use by cancer survivors: current use and future wishes","volume":"16","author":"van de Poll-Franse LV","year":"2008","unstructured":"van de Poll-Franse LV, van Eenbergen M. Internet use by cancer survivors: current use and future wishes. Support Care Cancer 2008;16:1189\u20131195."},{"key":"B4","series-title":"Health literacy: a manual for clinicians.","author":"Weiss BD","unstructured":"Weiss BD, Schwartzberg JG, Davis TC,  Health literacy: a manual for clinicians. Accessed October 5, 2023. Available at: http:\/\/lib.ncfh.org\/pdfs\/6617.pdf"},{"key":"B5","series-title":"J Health Care Poor Underserved","first-page":"99","article-title":"Illiteracy among Medicaid recipients and its relationship to health care costs","volume":"5","author":"Weiss BD","year":"1994","unstructured":"Weiss BD, Blanchard JS, McGee DL,  Illiteracy among Medicaid recipients and its relationship to health care costs. J Health Care Poor Underserved 1994;5:99\u2013111."},{"key":"B6","series-title":"JAMA","first-page":"1359","article-title":"Factors that prompted families to file medical malpractice claims following perinatal injuries","volume":"267","author":"Hickson GB","year":"1992","unstructured":"Hickson GB, Clayton EW, Githens PB,  Factors that prompted families to file medical malpractice claims following perinatal injuries. JAMA 1992;267:1359\u20131363."},{"key":"B7","series-title":"Lancet","first-page":"1609","article-title":"Why do people sue doctors? A study of patients and relatives taking legal action","volume":"343","author":"Vincent C","year":"1994","unstructured":"Vincent C, Young M, Phillips A. Why do people sue doctors? A study of patients and relatives taking legal action. Lancet 1994;343:1609\u20131613."},{"key":"B8","series-title":"JAMA","first-page":"475","article-title":"Association of health literacy with diabetes outcomes","volume":"288","author":"Schillinger D","year":"2002","unstructured":"Schillinger D, Grumbach K, Piette J,  Association of health literacy with diabetes outcomes. JAMA 2002;288:475\u2013482."},{"key":"B9","series-title":"Literacy","first-page":"24","article-title":"The concept of readability","volume":"2","author":"Gilliland J","year":"1968","unstructured":"Gilliland J. The concept of readability. Literacy 1968;2:24\u201329."},{"key":"B10","series-title":"J Public Health Manag Pract","first-page":"S235","article-title":"Healthy People 2030 leading health indicators and overall health and well-being measures: opportunities to assess and improve the health and well-being of the nation","volume":"27","author":"Ochiai E","year":"2021","unstructured":"Ochiai E, Kigenyi T, Sondik E,  Healthy People 2030 leading health indicators and overall health and well-being measures: opportunities to assess and improve the health and well-being of the nation. J Public Health Manag Pract 2021;27(Suppl 6):S235\u2013241."},{"key":"B11","article-title":"AHRQ health literacy universal precautions toolkit: 3rd edition","unstructured":"Brach C, ed. AHRQ health literacy universal precautions toolkit: 3rd edition. Accessed January 1, 2024. Available at: https:\/\/www.ahrq.gov\/sites\/default\/files\/publications2\/files\/health-literacy-universal-precautions-toolkit-3rd-edition.pdf"},{"key":"B12","series-title":"Ann Surg Oncol","first-page":"3898","article-title":"Readability and non- English language resources of hepatobiliary cancer center websites in the USA","volume":"30","author":"Murimwa GZ","year":"2023","unstructured":"Murimwa GZ, Stewart JW 2nd, Zhang L,  Readability and non- English language resources of hepatobiliary cancer center websites in the USA. Ann Surg Oncol 2023;30:3898\u20133900."},{"key":"B13","series-title":"Breast","first-page":"29","article-title":"Are English-language online patient education materials related to breast cancer risk assessment understandable, readable, and actionable?","volume":"61","author":"Lamb LR","year":"2022","unstructured":"Lamb LR, Baird GL, Roy IT,  Are English-language online patient education materials related to breast cancer risk assessment understandable, readable, and actionable? Breast 2022;61:29\u201334."},{"key":"B14","series-title":"J Cancer Educ","first-page":"1801","article-title":"How good is online information for patients on the treatment for luminal gastrointestinal cancers? A comprehensive evaluation in English and Spanish","volume":"38","author":"Beas R","year":"2023","unstructured":"Beas R, Cabanillas-Ramirez C, Izquierdo-Veraza D,  How good is online information for patients on the treatment for luminal gastrointestinal cancers? A comprehensive evaluation in English and Spanish. J Cancer Educ 2023;38:1801\u20131807."},{"key":"B15","series-title":"Nature","first-page":"20","article-title":"How Nature readers are using ChatGPT","volume":"615","author":"Owens B","year":"2023","unstructured":"Owens B. How Nature readers are using ChatGPT. Nature 2023;615:20."},{"key":"B16","year":"2023","unstructured":"OpenAI. ChatGPT; 2023. Accessed January 1, 2024. Available at: https:\/\/chat.openai.com\/chat"},{"key":"B17","series-title":"JAMA Intern Med","first-page":"589","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers JW","year":"2023","unstructured":"Ayers JW, Poliak A, Dredze M,  Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med 2023;183:589\u2013596."},{"key":"B18","series-title":"Urol Pract","first-page":"436","article-title":"Bridging the gap between urological research and patient understanding: the role of large language models in automated generation of layperson\u2019s summaries","volume":"10","author":"Eppler MB","year":"2023","unstructured":"Eppler MB, Ganjavi C, Knudsen JE,  Bridging the gap between urological research and patient understanding: the role of large language models in automated generation of layperson\u2019s summaries. Urol Pract 2023;10:436\u2013443."},{"key":"B19","series-title":"Biometrics","first-page":"159","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis JR","year":"1977","unstructured":"Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159\u2013174."},{"key":"B20","series-title":"J Read","article-title":"A readability formula that saves time","volume":"11","author":"Fry E","year":"1968","unstructured":"Fry E. A readability formula that saves time. J Read 1968;11:513\u2013516; 575\u2013578."},{"key":"B21","series-title":"J Appl Psychol","first-page":"221","article-title":"A new readability yardstick","volume":"32","author":"Flesch R","year":"1948","unstructured":"Flesch R. A new readability yardstick. J Appl Psychol 1948;32:221\u2013233."},{"key":"B22","series-title":"The Technique of Clear Writing.","author":"Gunning R","year":"1952","unstructured":"Gunning R. The Technique of Clear Writing. McGraw-Hill; 1952."},{"key":"B23","series-title":"J Read","first-page":"639","article-title":"SMOG grading\u2014a new readability formula","volume":"12","author":"McLaughlin GH","year":"1969","unstructured":"McLaughlin GH. SMOG grading\u2014a new readability formula. J Read 1969;12:639\u2013646."},{"key":"B24","series-title":"IEEE Transactions Prof Comm","first-page":"97","article-title":"How consistent are the best-known readability equations in estimating the readability of design standards","volume":"60","author":"Zhou S","year":"2017","unstructured":"Zhou S, Jeong H, Green PA. How consistent are the best-known readability equations in estimating the readability of design standards? IEEE Transactions Prof Comm 2017;60:97\u2013111."},{"key":"B25","series-title":"J Patient Exp","first-page":"2374373521998847","article-title":"Readability of patient education materials from high-impact medical journals: a 20-year analysis","volume":"8","author":"Rooney MK","year":"2021","unstructured":"Rooney MK, Santiago G, Perni S,  Readability of patient education materials from high-impact medical journals: a 20-year analysis. J Patient Exp 2021;8:2374373521998847."},{"key":"B26","series-title":"Pediatr Nurs","first-page":"274","article-title":"Assessing readability of patient information materials","volume":"9","author":"Spadero DC","year":"1983","unstructured":"Spadero DC. Assessing readability of patient information materials. Pediatr Nurs 1983;9:274\u2013278."},{"key":"B27","series-title":"Med Inform Internet Med","first-page":"67","article-title":"Health literacy and the world wide web: comparing the readability of leading incident cancers on the internet","volume":"31","author":"Friedman DB","year":"2006","unstructured":"Friedman DB, Hoffman-Goetz L, Arocha JF. Health literacy and the world wide web: comparing the readability of leading incident cancers on the internet. Med Inform Internet Med 2006;31:67\u201387."},{"key":"B28","series-title":"Health Educ Behav","first-page":"352","article-title":"A systematic review of readability and comprehension instruments used for print and web-based cancer information","volume":"33","author":"Friedman DB","year":"2006","unstructured":"Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav 2006;33:352\u2013373."},{"key":"B29","series-title":"Foot Ankle Int","first-page":"1282","article-title":"Improving the readability of online foot and ankle patient education materials","volume":"35","author":"Sheppard ED","year":"2014","unstructured":"Sheppard ED, Hyde Z, Florence MN,  Improving the readability of online foot and ankle patient education materials. Foot Ankle Int 2014;35:1282\u20131286."},{"key":"B30","series-title":"ANZ J Surg","first-page":"68","article-title":"The readability of original articles in surgical journals","volume":"76","author":"Hall JC","year":"2006","unstructured":"Hall JC. The readability of original articles in surgical journals. ANZ J Surg 2006;76:68\u201370."},{"key":"B31","series-title":"Inf Process Manage","first-page":"427","article-title":"A systematic analysis of performance measures for classification tasks","volume":"45","author":"Sokolova M","year":"2009","unstructured":"Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Inf Process Manage 2009;45:427\u2013437."},{"key":"B32","series-title":"IEEE Data Eng Bull","first-page":"35","article-title":"Modern information retrieval: a brief overview","volume":"24","author":"Singhal A","year":"2001","unstructured":"Singhal A. Modern information retrieval: a brief overview. IEEE Data Eng Bull 2001;24:35\u201343."},{"key":"B33","series-title":"Introduction to Data Mining, 1st ed","author":"Tan PN","year":"2005","unstructured":"Tan PN, Steinbach M, Kumar V. Introduction to Data Mining, 1st ed. Addison-Wesley; 2005"},{"key":"B34","series-title":"IEEE","first-page":"1","volume":"1","author":"Kadhim AI","year":"2014","unstructured":"Kadhim AI, Cheah YN, Ahamed NH,  Feature extraction for co-occurrence-based cosine similarity score of text documents. IEEE 2014;1:1\u20134."},{"key":"B35","series-title":"J Epidemiol Community Health","first-page":"105","article-title":"DISCERN: an instrument for judging the quality of written consumer health information on treatment choices","volume":"53","author":"Charnock D","year":"1999","unstructured":"Charnock D, Shepperd S, Needham G,  DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105\u2013111."},{"key":"B36","series-title":"Surgery","first-page":"30","article-title":"The quality and content of hyperthermic intraperitoneal chemotherapy information available to patients: an evaluation of North American hyperthermic intraperitoneal chemotherapy websites","volume":"174","author":"Remer SL","year":"2023","unstructured":"Remer SL, Connelly TM, Clancy C,  The quality and content of hyperthermic intraperitoneal chemotherapy information available to patients: an evaluation of North American hyperthermic intraperitoneal chemotherapy websites. Surgery 2023;174:30\u201335."},{"key":"B37","series-title":"Prostate Cancer Prostatic Dis","first-page":"103","article-title":"Quality of information and appropriateness of ChatGPT outputs for urology patients","volume":"27","author":"Cocci A","year":"2024","unstructured":"Cocci A, Pezzoli M, Lo Re M,  Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer Prostatic Dis 2024;27:103\u2013108."},{"key":"B38","author":"Goodman M","unstructured":"Goodman M, Finnegan R, Mohadjer L,  Literacy, numeracy, and problem solving in technology-rich environments among U.S. adults: results from the Program for the International Assessment of Adult Competencies 2012. Accessed January 1, 2024. Available at: https:\/\/nces.ed.gov\/pubs2014\/2014008.pdf"},{"key":"B39","series-title":"Urology","first-page":"35","article-title":"Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer?","volume":"180","author":"Coskun B","year":"2023","unstructured":"Coskun B, Ocakoglu G, Yetemen M,  Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer? Urology 2023;180:35\u201358."},{"key":"B40","series-title":"Eur J Cardiovasc Nurs","article-title":"What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value and danger of ChatGPT-generated responses to health questions","author":"Van Bulck L","unstructured":"Van Bulck L, Moons P. What if your patient switches from Dr. Google to Dr. ChatGPT? A vignette-based survey of the trustworthiness, value and danger of ChatGPT-generated responses to health questions. Eur J Cardiovasc Nurs 2024;23:95\u201398."},{"key":"B41","series-title":"JAMA Oncol","first-page":"1459","article-title":"Use of artificial intelligence chatbots for cancer treatment information","volume":"9","author":"Chen S","year":"2023","unstructured":"Chen S, Kann BH, Foote MB,  Use of artificial intelligence chatbots for cancer treatment information. JAMA Oncol 2023;9:1459\u20131462."},{"key":"B42","series-title":"JAMA Surg","first-page":"137","article-title":"Association of health literacy with postoperative outcomes in patients undergoing major abdominal surgery","volume":"153","author":"Wright JP","year":"2018","unstructured":"Wright JP, Edwards GC, Goggins K,  Association of health literacy with postoperative outcomes in patients undergoing major abdominal surgery. JAMA Surg 2018;153:137\u2013142."},{"key":"B43","series-title":"Cancer","first-page":"1912","article-title":"Knowledge and attitude on screening mammography among low-literate, low-income women","volume":"78","author":"Davis TC","year":"1996","unstructured":"Davis TC, Arnold C, Berkel HJ,  Knowledge and attitude on screening mammography among low-literate, low-income women. Cancer 1996;78:1912\u20131920."},{"key":"B44","series-title":"JAMA","first-page":"552","article-title":"Clinical, legal, and ethical aspects of artificial intelligence-assisted conversational agents in health care","volume":"324","author":"McGreevey JD 3rd","year":"2020","unstructured":"McGreevey JD 3rd, Hanson CW 3rd, Koppel R. Clinical, legal, and ethical aspects of artificial intelligence-assisted conversational agents in health care. JAMA 2020;324:552\u2013553."},{"key":"B45","series-title":"Nat Rev Urol","article-title":"Artificial intelligence and urology: ethical considerations for urologists and patients","author":"Cacciamani GE","unstructured":"Cacciamani GE, Chen A, Gill IS,  Artificial intelligence and urology: ethical considerations for urologists and patients. Nat Rev Urol 2024;21:50\u201359."},{"key":"B46","unstructured":"US National Cancer Institute. NCI-designated cancer centers. Accessed January 1, 2024. Available at: https:\/\/www.cancer.gov\/research\/infrastructure\/cancer-centers"}],"container-title":["Journal of the National Comprehensive Cancer Network"],"original-title":[],"link":[{"URL":"https:\/\/jnccn.org\/view\/journals\/jnccn\/22\/2D\/article-e237334.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/jnccn.org\/downloadpdf\/journals\/jnccn\/22\/2D\/article-e237334.xml","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/jnccn.org\/downloadpdf\/journals\/jnccn\/22\/2D\/article-e237334.xml","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,11]],"date-time":"2024-06-11T19:49:04Z","timestamp":1718135344000},"score":1,"resource":{"primary":{"URL":"https:\/\/jnccn.org\/view\/journals\/jnccn\/22\/2D\/article-e237334.xml"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6]]},"references-count":46,"journal-issue":{"issue":"2D"},"URL":"https:\/\/doi.org\/10.6004\/jnccn.2023.7334","relation":{},"ISSN":["1540-1405","1540-1413"],"issn-type":[{"value":"1540-1405","type":"print"},{"value":"1540-1413","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6]]}}}