{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T02:23:58Z","timestamp":1793327038279,"version":"4.5.0"},"reference-count":33,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2025,8,12]],"date-time":"2025-08-12T00:00:00Z","timestamp":1754956800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: Large language models (LLMs) such as ChatGPT, Google Gemini, and Microsoft Copilot are increasingly used by patients seeking medical information online. While these tools provide accessible and conversational explanations, their accuracy and safety in emotionally sensitive scenarios\u2014such as an incidental cancer diagnosis\u2014remain uncertain. Objective: To evaluate the quality, completeness, readability, and safety of responses generated by three state-of-the-art LLMs to common patient questions following the incidental discovery of a kidney tumor. Methods: A standardized use-case scenario was developed: a patient learns of a suspicious renal mass following a computed tomography (CT) scan for back pain. Ten plain-language prompts reflecting typical patient concerns were submitted to ChatGPT-4o, Microsoft Copilot, and Google Gemini 2.5 Pro without additional context. Responses were independently assessed by five board-certified urologists using a validated six-domain rubric (accuracy, completeness, clarity, currency, risk of harm, hallucinations), scored on a 1\u20135 Likert scale. Two statistical approaches were applied to calculate descriptive scores and inter-rater reliability (Fleiss\u2019 Kappa). Readability was analyzed using the Flesch Reading Ease (FRE) and Flesch\u2013Kincaid Grade Level (FKGL) metrics. Results: Google Gemini 2.5 Pro achieved the highest mean ratings across most domains, notably in accuracy (4.3), completeness (4.3), and low hallucination rate (4.6). Microsoft Copilot was noted for empathetic language and consistent disclaimers but showed slightly lower clarity and currency scores. ChatGPT-4o demonstrated strengths in conversational flow but displayed more variability in clinical precision. Overall, 14% of responses were flagged as potentially misleading or incomplete. Inter-rater agreement was substantial across all domains (\u03ba = 0.68). Readability varied between models: ChatGPT responses were easiest to understand (FRE = 48.5; FKGL = 11.94), while Gemini\u2019s were the most complex (FRE = 29.9; FKGL = 13.3). Conclusions: LLMs show promise in patient-facing communication but currently fall short of providing consistently accurate, complete, and guideline-conform information in high-stakes contexts such as incidental cancer diagnoses. While their tone and structure may support patient engagement, they should not be used autonomously for counseling. Further fine-tuning, clinical validation, and supervision are essential for safe integration into patient care.<\/jats:p>","DOI":"10.3390\/jcm14165697","type":"journal-article","created":{"date-parts":[[2025,8,12]],"date-time":"2025-08-12T16:30:36Z","timestamp":1755016236000},"page":"5697","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["How Accurate Is AI? A Critical Evaluation of Commonly Used Large Language Models in Responding to Patient Concerns About Incidental Kidney Tumors"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7226-3852","authenticated-orcid":false,"given":"Bernhard","family":"Ralla","sequence":"first","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nadine","family":"Biernath","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Isabel","family":"Lichy","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lukas","family":"Kurz","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank","family":"Friedersdorff","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thorsten","family":"Schlomm","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-0319-5002","authenticated-orcid":false,"given":"Jacob","family":"Schmidt","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7680-1870","authenticated-orcid":false,"given":"Henning","family":"Plage","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7035-9127","authenticated-orcid":false,"given":"Jonathan","family":"Jeutner","sequence":"additional","affiliation":[{"name":"Department of Urology, Charit\u00e9\u2013Universit\u00e4tsmedizin Berlin, Corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin, Charit\u00e9platz 1, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e64290","DOI":"10.2196\/64290","article-title":"Laypeople\u2019s Use of and Attitudes Toward Large Language Models and Sear ch Engines for Health Queries: Survey Study","volume":"27","author":"Mendel","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e67383","DOI":"10.2196\/67383","article-title":"Perspectives and Experiences with Large Language Models in Health Care: Survey Study","volume":"27","author":"Sumner","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Reuter, N., von Lipinski, V.-N., Jeutner, J., Schlomm, T., Witzenrath, M., Sander, L.E., and Gr\u00f6schel, M.I. (2025). AI-generated patient-friendly discharge summaries to empower patients. medRxiv.","DOI":"10.1101\/2025.07.04.25330804"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Rodler, S., Cei, F., Ganjavi, C., Checcucci, E., De Backer, P., Belenchon, I.R., Taratkin, M., Puliatti, S., Veccia, A., and Piazza, P. (2025). GPT-4 generates accurate and readable patient education materials aligned with current oncological guidelines: A randomized assessment. PLoS ONE, 20.","DOI":"10.1371\/journal.pone.0324175"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e2457879","DOI":"10.1001\/jamanetworkopen.2024.57879","article-title":"Large Language Models for Chatbot Health Advice Studies","volume":"8","author":"Huo","year":"2025","journal-title":"JAMA Netw. Open"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Aydin, S., Karabacak, M., Vlachos, V., and Margetis, K. (2024). Large language models in patient education: A scoping review of applic ations in medicine. Front. Med., 11.","DOI":"10.3389\/fmed.2024.1477898"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1186\/s12943-024-02140-6","article-title":"Artificial intelligence alphafold model for molecular biology and drug discovery: A machine-learning-driven informatics investigation","volume":"23","author":"Guo","year":"2024","journal-title":"Mol. Cancer"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wen, B., Norel, R., Liu, J., Stappenbeck, T., Zulkernine, F., and Chen, H. (2024, January 7\u201313). Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Health. Proceedings of the 2024 IEEE International Conference on Digital Health (ICDH), Shenzhen, China.","DOI":"10.1109\/ICDH62654.2024.00027"},{"key":"ref_9","unstructured":"Mannhardt, N., Bondi-Kelly, E., Lam, B., Mozannar, H., O\u2019Connell, C., Asiedu, M., Buendia, A., Urman, T., Riaz, I.B., and Ricciardi, C.E. (2024). Impact of Large Language Model Assistance on Patients Reading Clinical Notes: A Mixed-Methods Study. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"ooaf021","DOI":"10.1093\/jamiaopen\/ooaf021","article-title":"A proof-of-concept study for patient use of open notes with large lang uage models","volume":"8","author":"Salmi","year":"2025","journal-title":"JAMIA Open"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.36922\/aih.4808","article-title":"Role of large language models in improving provider\u2013patient experience and interaction efficiency: A scoping review","volume":"2","author":"Vishwanath","year":"2024","journal-title":"Artif. Intell. Health"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1038\/s43856-024-00717-2","article-title":"Current applications and challenges in large language models for patie nt care: A systematic review","volume":"5","author":"Busch","year":"2025","journal-title":"Commun. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1186\/s13000-024-01464-7","article-title":"Challenges and barriers of using large language models (LLM) such as C hatGPT for diagnostic medicine with a focus on digital pathology\u2014A recent scoping review","volume":"19","author":"Ullah","year":"2024","journal-title":"Diagn. Pathol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and Evaluation of Health Care Applications of Large Language M odels","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"ref_15","unstructured":"Umerenkov, D., Zubkova, G., and Nesterov, A. (2023). Deciphering Diagnoses: How Large Language Models Explanations Influenc e Clinical Decision Making. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"100085","DOI":"10.1016\/j.apjo.2024.100085","article-title":"Understanding natural language: Potential application of large languag e models to ophthalmology","volume":"13","author":"Yang","year":"2024","journal-title":"Asia-Pac. J. Ophthalmol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1016\/j.eururo.2025.02.020","article-title":"European Association of Urology Guidelines on Renal Cell Carcinoma: The 2025 Update","volume":"87","author":"Bex","year":"2025","journal-title":"Eur. Urol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1016\/S0090-4295(03)00658-7","article-title":"Histopathology of surgically managed renal tumors: Analysis of a conte mporary series","volume":"62","author":"Duchene","year":"2003","journal-title":"Urology"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1016\/j.urology.2013.01.009","article-title":"A Review of Contemporary Data on Surgically Resected Renal Masses\u2014Beni gn or Malignant?","volume":"81","author":"Corcoran","year":"2013","journal-title":"Urology"},{"key":"ref_20","first-page":"A4932","article-title":"High rate of benign histology in radiologically suspect renal lesions","volume":"61","author":"Pedersen","year":"2014","journal-title":"Dan. Med. J."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1200\/jco.2012.30.5_suppl.357","article-title":"Incidence of benign versus malignant renal tumors in selected studies","volume":"30","author":"Russo","year":"2012","journal-title":"J. Clin. Oncol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"79","DOI":"10.3233\/KCA-180028","article-title":"Diagnostic Imaging for Solid Renal Tumors: A Pictorial Review","volume":"2","author":"Mulders","year":"2018","journal-title":"Kidney Cancer"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"575","DOI":"10.2214\/ajr.183.3.1830575","article-title":"Evaluation of Patients Referred for Percutaneous Ablation of Renal Tum ors: Importance of a Preprocedural Diagnosis","volume":"183","author":"Tuncali","year":"2004","journal-title":"Am. J. Roentgenol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1007\/s00345-024-05043-9","article-title":"Three-dimensional virtual model for robot-assisted partial nephrectomy: A propensity-score matching analysis with a contemporary control group","volume":"42","author":"Grosso","year":"2024","journal-title":"World J. Urol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Geant\u0103, M., B\u0103descu, D., Chirca, N., Nechita, O.C., Radu, C.G., Rascu, S., R\u0103d\u0103voi, D., Sima, C., Toma, C., and Jinga, V. (2024). The Potential Impact of Large Language Models on Doctor\u2013Patient Commun ication: A Case Study in Prostate Cancer. Healthcare, 12.","DOI":"10.3390\/healthcare12151548"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhu, L., Mou, W., and Chen, R. (2023). Can the ChatGPT and other Large Language Models with internet-connecte d database solve the questions and concerns of patient with prostate c ancer?. medRxiv.","DOI":"10.1101\/2023.03.06.23286827"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1007\/s00066-024-02342-3","article-title":"Patient- and clinician-based evaluation of large language models for p atient education in prostate cancer radiotherapy","volume":"201","author":"Trapp","year":"2025","journal-title":"Strahlenther. Und Onkol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Alasker, A., Alshathri, N., Alsalamah, S., Almansour, N., Alsalamah, F., Alghafees, M., AlKhamees, M., and Alsaikhan, B. (2025). ChatGPT vs. Gemini: Which Provides Better Information on Bladder Cancer. Soci\u00e9t\u00e9 Int. D\u2019urologie J., 6.","DOI":"10.3390\/siuj6020034"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"843","DOI":"10.3390\/siuj5060064","article-title":"Utility of ChatGPT and Large Language Models in Enhancing Patient Unde rstanding of Urological Conditions","volume":"5","author":"Mak","year":"2024","journal-title":"Soci\u00e9t\u00e9 Int. D\u2019urologie J."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1010","DOI":"10.1111\/bju.16676","article-title":"Evaluating interactions of patients with large language models for medical information","volume":"135","author":"Carl","year":"2025","journal-title":"BJU Int."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1002\/bco2.359","article-title":"Urology consultants versus large language models: Potentials and hazards for medical advice in urology","volume":"5","author":"Eckrich","year":"2024","journal-title":"BJUI Compass"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"e107","DOI":"10.1097\/ICL.0000000000001158","article-title":"Evaluation of Responses to Questions About Keratoconus Using ChatGPT-4.0, Google Gemini and Microsoft Copilot: A Comparative Study of Large Language Models on Keratoconus","volume":"51","author":"Demir","year":"2024","journal-title":"Eye Contact Lens Sci. Clin. Pract."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"706","DOI":"10.1186\/s12967-025-06751-5","article-title":"Surge in large language models exacerbates global regional healthcare inequalities","volume":"23","author":"Guo","year":"2025","journal-title":"J. Transl. Med."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/14\/16\/5697\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:25:25Z","timestamp":1760034325000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/14\/16\/5697"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,12]]},"references-count":33,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["jcm14165697"],"URL":"https:\/\/doi.org\/10.3390\/jcm14165697","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,12]]}}}