{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T10:48:28Z","timestamp":1789814908888,"version":"4.0.1"},"reference-count":42,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T00:00:00Z","timestamp":1752796800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background\/Objectives: This study evaluates the performance of a large language model (lLm) in providing geographically tailored colorectal cancer screening and surveillance recommendations to gastrointestinal surgeons. Methods: Fifty-four patient cases, varying by age and family history, were developed based on colorectal cancer guidelines. Standardized prompts with predefined query terms were used to query ChatGPT-4.5 on 18 April 2025, from four locations: Canada, Italy, Romania, and the United Kingdom. Responses were classified as \u201cCorrect,\u201d \u201cPartially Correct,\u201d or \u201cIncorrect\u201d based on clinical guidelines and expert recommendations for each country. Outcomes were analyzed using descriptive statistics. Results: ChatGPT provided recommendations on screening eligibility, test interpretation, the management of positive results, and surveillance intervals. Correct recommendations were given for 50.0% (27\/54) of cases in Canada, 63.0% (34\/54) of cases in Italy, 40.7% (22\/54) of cases in Romania, and 55.6% (30\/54) of cases in the United Kingdom. Queries in Italian yielded correct guidance for 64.8% (35\/54) of cases, while Romanian queries were accurate for 40.7% (22\/54) of cases. Notably, Romania and Italy lacked detailed guidelines for polyp management and post-test surveillance. A key finding was the inconsistency between ChatGPT-generated titles and corresponding recommendations, which may impact its reliability in clinical decision-making. Conclusions: ChatGPT-4.5\u2019s performance varies by country and language, highlighting inconsistencies in geographically tailored recommendations. This study highlights limitations associated with the training data cutoff and the potential biases introduced by model-generated responses. Healthcare professionals should recognize these limitations and the possible gaps in guideline availability, particularly for high-risk screening, polyp management, and surveillance in certain European countries.<\/jats:p>","DOI":"10.3390\/jcm14145101","type":"journal-article","created":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T11:12:34Z","timestamp":1752837154000},"page":"5101","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Dr. LLM Will See You Now: The Ability of ChatGPT to Provide Geographically Tailored Colorectal Cancer Screening and Surveillance Recommendations"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1138-9349","authenticated-orcid":false,"given":"Aisling","family":"Zeng","sequence":"first","affiliation":[{"name":"Michael G. DeGroote School of Medicine, McMaster University, Hamilton, ON L8S 4L8, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacqueline","family":"Steinke","sequence":"additional","affiliation":[{"name":"Division of General\/Colorectal Surgery, Ashford and St. Peter\u2019s Hospitals NHS Foundation Trust, Chertsey KT16 0PZ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0592-2034","authenticated-orcid":false,"given":"Horea-Florin","family":"Bocse","sequence":"additional","affiliation":[{"name":"Department of General Surgery, Regional Institute of Gastroenterology and Hepatology, 400162 Cluj-Napoca, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5244-4357","authenticated-orcid":false,"given":"Matteo","family":"De Pastena","sequence":"additional","affiliation":[{"name":"Department of General and Pancreatic Surgery, The Pancreas Institute, University of Verona Hospital Trust, 37129 Verona, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N. Engl. J. Med."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Dave, T., Athaluri, S.A., and Singh, S. (2023). ChatGPT in medicine: An overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front. Artif. Intell., 6.","DOI":"10.3389\/frai.2023.1169595"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Temsah, M.-H., Aljamaan, F., Malki, K.H., Alhasan, K., Altamimi, I., Aljarbou, R., Bazuhair, F., Alsubaihin, A., Abdulmajeed, N., and Alshahrani, F.S. (2023). ChatGPT and the Future of Digital Health: A Study on Healthcare Workers\u2019 Perceptions and Expectations. Healthcare, 11.","DOI":"10.3390\/healthcare11131812"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"210","DOI":"10.7326\/M23-2772","article-title":"Large Language Models in Medicine: The Potentials and Pitfalls: A Narrative Review","volume":"177","author":"Omiye","year":"2024","journal-title":"Ann. Intern. Med."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Gallifant, J., Fiske, A., Strekalova, Y.A., Osorio-Valencia, J.S., Parke, R., Mwavu, R., Martinez, N., Gichoya, J.W., Ghassemi, M., and Demner-Fushman, D. (2024). Peer review of GPT-4 technical report and systems card. PLOS Digit. Health, 3.","DOI":"10.1371\/journal.pdig.0000417"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e48009","DOI":"10.2196\/48009","article-title":"Ethical Considerations of Using ChatGPT in Health Care","volume":"25","author":"Wang","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Naik, N., Hameed, B.M.Z., Shetty, D.K., Swain, D., Shah, M., Paul, R., Aggarwal, K., Ibrahim, S., Patil, V., and Smriti, K. (2022). Legal and Ethical Consideration in Artificial Intelligence in Healthcare: Who Takes Responsibility?. Front. Surg., 9.","DOI":"10.3389\/fsurg.2022.862322"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","year":"2023","journal-title":"Commun. Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1007\/s12553-024-00836-9","article-title":"Dr. GPT will see you now: The ability of large language model-linked chatbots to provide colorectal cancer screening recommendations","volume":"14","author":"Huo","year":"2024","journal-title":"Health Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1022","DOI":"10.1097\/MCG.0000000000001979","article-title":"Colorectal Cancer Prevention: Is Chat Generative Pretrained Transformer (Chat GPT) ready to Assist Physicians in Determining Appropriate Screening and Surveillance Recommendations?","volume":"58","author":"Pereyra","year":"2024","journal-title":"J. Clin. Gastroenterol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1111\/ans.18749","article-title":"Conversational artificial intelligence (chatGPTTM) in the management of complex colorectal cancer patients: Early experience","volume":"94","author":"Choo","year":"2024","journal-title":"ANZ J. Surg."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"6161","DOI":"10.1007\/s00405-024-08708-8","article-title":"The utility of ChatGPT as a generative medical translator","volume":"281","author":"Grimm","year":"2024","journal-title":"Eur. Arch. Otorhinolaryngol."},{"key":"ref_13","unstructured":"Li, Z., Shi, Y., Liu, Z., Yang, F., Liu, N., and Du, M. (2024). Quantifying Multilingual Performance of Large Language Models Across Languages. arXiv."},{"key":"ref_14","first-page":"a25689416","article-title":"Exploring ChatGPT effectiveness in addressing direct patient queries on colorectal cancer screening","volume":"13","author":"Maida","year":"2025","journal-title":"Endosc. Int. Open"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1370\/afm.104","article-title":"Designing a mixed methods study in primary care","volume":"2","author":"Creswell","year":"2004","journal-title":"Ann. Fam. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"2134","DOI":"10.1111\/1475-6773.12117","article-title":"Achieving integration in mixed methods designs-principles and practices","volume":"48","author":"Fetters","year":"2013","journal-title":"Health Serv. Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1177\/1558689819875832","article-title":"A Checklist of Mixed Methods Elements in a Submission for Advancing the Methodology of Mixed Methods Research","volume":"13","author":"Fetters","year":"2019","journal-title":"J. Mix. Methods Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1038\/s41591-022-01772-9","article-title":"Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"28","author":"Vasey","year":"2022","journal-title":"Nat. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"250","DOI":"10.3322\/caac.21457","article-title":"Colorectal cancer screening for average-risk adults: 2018 guideline update from the American Cancer Society","volume":"68","author":"Wolf","year":"2018","journal-title":"CA Cancer J. Clin."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.1038\/ajg.2017.174","article-title":"Colorectal Cancer Screening: Recommendations for Physicians and Patients from the U.S. Multi-Society Task Force on Colorectal Cancer","volume":"112","author":"Rex","year":"2017","journal-title":"Am. J. Gastroenterol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1136\/gutjnl-2019-319858","article-title":"British Society of Gastroenterology\/Association of Coloproctology of Great Britain and Ireland\/Public Health England post-polypectomy and post-colorectal cancer resection surveillance guidelines","volume":"69","author":"Rutter","year":"2020","journal-title":"Gut"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2988","DOI":"10.1038\/s41591-023-02656-2","article-title":"Reporting standards for the use of large language model-linked chatbots for health advice","volume":"29","author":"Huo","year":"2023","journal-title":"Nat. Med."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1016\/j.jclinepi.2010.07.015","article-title":"GRADE guidelines: 3. Rating the quality of evidence","volume":"64","author":"Balshem","year":"2011","journal-title":"J. Clin. Epidemiol."},{"key":"ref_24","first-page":"i2016","article-title":"GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 1: Introduction","volume":"353","author":"Moberg","year":"2016","journal-title":"BMJ"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1136\/bmj.318.7183.593","article-title":"Clinical guidelines: Developing guidelines","volume":"318","author":"Shekelle","year":"1999","journal-title":"BMJ"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1016\/j.jclinepi.2013.02.003","article-title":"GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation\u2019s direction and strength","volume":"66","author":"Andrews","year":"2013","journal-title":"J. Clin. Epidemiol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"183","DOI":"10.34172\/hpp.2023.22","article-title":"Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: A systematic review","volume":"13","author":"Garg","year":"2023","journal-title":"Health Promot. Perspect."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1109\/JAS.2023.123618","article-title":"A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development","volume":"10","author":"Wu","year":"2023","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_29","unstructured":"Ateia, S., and Kruschwitz, U. (2023). Is ChatGPT a Biomedical Expert?\u2014Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks. arXiv."},{"key":"ref_30","unstructured":"Garcia, X., Bansal, Y., Cherry, C., Foster, G., Krikun, M., Feng, F., Johnson, M., and Firat, O. (2023). The unreasonable effectiveness of few-shot learning for machine translation. arXiv."},{"key":"ref_31","unstructured":"Lample, G., Conneau, A., Denoyer, L., and Ranzato, M. (2018). Unsupervised Machine Translation Using Monolingual Corpora Only. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.iotcps.2023.04.003","article-title":"ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope","volume":"3","author":"Ray","year":"2023","journal-title":"Int. Things Cyber-Phys. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e48002","DOI":"10.2196\/48002","article-title":"Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study","volume":"9","author":"Takagi","year":"2023","journal-title":"JMIR Med. Educ."},{"key":"ref_34","unstructured":"Nori, H., King, N., McKinney, S.M., Carignan, D., and Horvitz, E. (2023). Capabilities of GPT-4 on Medical Challenge Problems. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Brin, D., Sorin, V., Vaid, A., Soroush, A., Glicksberg, B.S., Charney, A.W., Nadkarni, G., and Klang, E. (2023). Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci. Rep., 13.","DOI":"10.1038\/s41598-023-43436-9"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1111\/j.1471-1842.2007.00713.x","article-title":"The information-seeking behaviour of doctors: A review of the evidence","volume":"24","author":"Davies","year":"2007","journal-title":"Health Info. Libr. J."},{"key":"ref_37","first-page":"55","article-title":"How do primary care physicians seek answers to clinical questions? A literature review","volume":"94","author":"Coumou","year":"2006","journal-title":"J. Med. Libr. Assoc."},{"key":"ref_38","unstructured":"OECD, and World Health Organization (2019). Improving Healthcare Quality in Europe: Characteristics, Effectiveness and Implementation of Different Strategies, OECD."},{"key":"ref_39","first-page":"340","article-title":"Canadian Task Force on Preventive Health Care","volume":"188","author":"Bacchus","year":"2016","journal-title":"CMAJ"},{"key":"ref_40","unstructured":"Cancer Care Ontario (2025, July 15). ColonCancerCheck Guidelines. Available online: https:\/\/www.cancercareontario.ca."},{"key":"ref_41","unstructured":"NHS England (2025, July 15). Bowel Cancer Screening Guide, Available online: https:\/\/www.gov.uk."},{"key":"ref_42","first-page":"1","article-title":"Characteristics of the colorectal cancers diagnosed in the early 2000s in Italy. Figures from the IMPATTO study on colorectal\ncancer screening","volume":"39","author":"Zorzi","year":"2015","journal-title":"Epidemiol Prev."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/14\/14\/5101\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:11:46Z","timestamp":1760033506000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/14\/14\/5101"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,18]]},"references-count":42,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["jcm14145101"],"URL":"https:\/\/doi.org\/10.3390\/jcm14145101","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,18]]}}}