{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:18:54Z","timestamp":1791458334646,"version":"4.3.4"},"reference-count":29,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Prostate-specific antigen (PSA) testing remains the cornerstone of early prostate cancer detection. Society guidelines for prostate cancer screening via PSA testing serve to standardize patient care and are often used by trainees, junior staff, or generalist medical practitioners to guide medical decision-making. However, adherence to guidelines is a time-consuming and challenging task, and rates of inappropriate PSA testing are high. Retrieval-augmented generation (RAG) is a method to enhance the reliability of large language models (LLMs) by grounding responses in trusted external sources.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate a RAG-enhanced LLM system, grounded in current European Association of Urology and American Urological Association guidelines, to assess its effectiveness in providing guideline-concordant PSA screening recommendations compared to junior clinicians.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A series of 44 fictional outpatient case scenarios was developed to represent a broad spectrum of clinical presentations. A RAG pipeline was developed, comprising a life expectancy estimation module based on the Charlson Comorbidity Index, followed by LLM-generated recommendations constrained to retrieved excerpts from the European Association of Urology and American Urological Association guidelines. Five junior clinicians were tasked to provide PSA testing recommendations for the same scenarios in closed-book and open-book formats. Answers were compared for accuracy in a binomial fashion. Fleiss \u03ba was computed to assess interrater agreement among clinicians.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The RAG-LLM tool provided guideline-concordant recommendations in 95.5% (210\/220) of case scenarios, compared to junior clinicians, who were correct in 62.3% (220) of scenarios in a closed-book format and 74.1% (220) of scenarios in an open-book format. The difference was statistically significant for both closed-book (\n P<\/jats:italic>\n <.001) and open-book (\n P<\/jats:italic>\n <.001) formats. Interrater agreement among clinicians was fair, with Fleiss \u03ba of 0.294 and 0.321 for closed-book and open-book formats, respectively.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Use of RAG techniques allows LLMs to integrate complex guidelines into day-to-day medical decision-making. RAG-LLM tools in urology have the capability to enhance clinical decision-making by providing guideline-concordant recommendations for PSA testing, potentially improving the consistency of health care delivery, reducing cognitive load on clinicians, and reducing unnecessary investigations and costs. While this study used synthetic cases in a controlled simulation environment, it establishes a foundation for future validation in real-world clinical settings.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/78393","type":"journal-article","created":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T15:05:08Z","timestamp":1760281508000},"page":"e78393-e78393","source":"Crossref","is-referenced-by-count":14,"title":["Performance of Retrieval-Augmented Generation Large Language Models in Guideline-Concordant Prostate-Specific Antigen Testing: Comparative Study With Junior Clinicians"],"prefix":"10.2196","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4991-9790","authenticated-orcid":false,"given":"Joshua Yi Min","family":"Tung","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3306-3000","authenticated-orcid":false,"given":"Quan","family":"Le","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7499-9134","authenticated-orcid":false,"given":"Jinxuan","family":"Yao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8907-7198","authenticated-orcid":false,"given":"Yifei","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9715-6970","authenticated-orcid":false,"given":"Daniel Yan Zheng","family":"Lim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8795-7491","authenticated-orcid":false,"given":"Gerald Gui Ren","family":"Sng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4326-0465","authenticated-orcid":false,"given":"Rachel Shu En","family":"Lau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8166-0582","authenticated-orcid":false,"given":"Yu Guang","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3844-7567","authenticated-orcid":false,"given":"Kenneth","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0365-1899","authenticated-orcid":false,"given":"Kae Jack","family":"Tay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7785-2987","authenticated-orcid":false,"given":"Jen Hong","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6367-2563","authenticated-orcid":false,"given":"John Shyi Peng","family":"Yuen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5274-3004","authenticated-orcid":false,"given":"Christopher Wai Sam","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9072-9742","authenticated-orcid":false,"given":"Henry Sun Sien","family":"Ho","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,11,19]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"229","DOI":"10.3322\/caac.21834","article-title":"Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"74","author":"Bray","journal-title":"CA A Cancer J Clinicians"},{"issue":"13","key":"R2","doi-asserted-by":"publisher","first-page":"981","DOI":"10.1093\/jnci\/94.13.981","article-title":"Overdiagnosis due to prostate-specific antigen screening: lessons from U.S. prostate cancer incidence trends","volume":"94","author":"Etzioni","journal-title":"J Natl Cancer Inst"},{"issue":"2","key":"R3","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1111\/bju.12368","article-title":"Mortality and complications after prostate biopsy in the Prostate, Lung, Colorectal and Ovarian cancer screening (PLCO) trial","volume":"113","author":"Pinsky","journal-title":"BJU Int"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1016\/j.eururo.2024.03.027","article-title":"EAU-EANM-ESTRO-ESUR-ISUP-SIOG guidelines on prostate cancer-2024 update. Part I: screening, diagnosis, and local treatment with curative intent","volume":"86","author":"Cornford","journal-title":"Eur Urol"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1097\/JU.0000000000003491","article-title":"Early detection of prostate cancer: AUA\/SUO guideline part I: prostate cancer screening","volume":"210","author":"Wei","journal-title":"J Urol"},{"issue":"4","key":"R6","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.7504","article-title":"Low-value prostate-specific antigen screening in older males","volume":"6","author":"Kalavacherla","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","journal-title":"Commun Med"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/58041","article-title":"Enhancement of the performance of large language models in diabetes education through retrieval-augmented generation: comparative study","volume":"26","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1093\/jamia\/ocaf008","article-title":"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines","volume":"32","author":"Liu","journal-title":"J Am Med Inform Assoc"},{"issue":"103078","key":"R10","doi-asserted-by":"publisher","first-page":"103078","DOI":"10.1016\/j.artmed.2025.103078","article-title":"Empowering large language models for automated clinical assessment with generation-augmented retrieval and hierarchical chain-of-thought","volume":"162","author":"Gu","journal-title":"Artif Intell Med"},{"issue":"5","key":"R11","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1016\/0021-9681(87)90171-8","article-title":"A new method of classifying prognostic comorbidity in longitudinal studies: development and validation","volume":"40","author":"Charlson","journal-title":"J Chronic Dis"},{"key":"R12","unstructured":"Death and life expectancy. Statistics Singapore. URL: http:\/\/www.singstat.gov.sg\/find-data\/search-by-theme\/population\/death-and-life-expectancy\/latest-data [Accessed 30-09-2024]"},{"key":"R13","unstructured":"Wei J Wang X Schuurmans D et al. Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo S Mohamed S Agarwal A Belgrave D , editors. NIPS \u201922: Proceedings of the 36th International Conference on Neural Information Processing Systems. Curran Associates Inc; 2024:24824-24837. ISBN: 9781713871088"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1111\/jgh.16375","article-title":"ChatGPT on guidelines: providing contextual knowledge to GPT allows it to provide advice on appropriate colonoscopy intervals","volume":"39","author":"Lim","journal-title":"J Gastroenterol Hepatol"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1101\/2023.11.10.23298364","article-title":"Development of a liver disease-specific large language model chat interface using retrieval augmented generation","author":"Ge","journal-title":"Gastroenterology"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","DOI":"10.1136\/openhrt-2023-002432","article-title":"Harnessing the power of clinical decision support systems: challenges and opportunities","volume":"10","author":"Chen","journal-title":"Open Heart"},{"issue":"629","key":"R17","doi-asserted-by":"publisher","first-page":"e765","DOI":"10.3399\/bjgp14X682837","article-title":"Referral interventions from primary to specialist care: a systematic review of international evidence","volume":"64","author":"Blank","journal-title":"Br J Gen Pract"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01081-0","article-title":"Augmented non-hallucinating large language models as medical information curators","volume":"7","author":"Gilbert","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-023-04079-6","article-title":"The benefits of using atypical presentations and rare diseases in problem-based learning in undergraduate medical education","volume":"23","author":"Bai","journal-title":"BMC Med Educ"},{"issue":"10","key":"R20","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.1111\/medu.13367","article-title":"Exploring examinee behaviours as validity evidence for multiple-choice question examinations","volume":"51","author":"Surry","journal-title":"Med Educ"},{"issue":"10","key":"R21","doi-asserted-by":"publisher","first-page":"1545","DOI":"10.1097\/ACM.0b013e3182a31c1e","article-title":"Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought","volume":"88","author":"Ilgen","journal-title":"Acad Med"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2022.03.022","article-title":"The Charlson Comorbidity Index: problems with use in epidemiological research","volume":"148","author":"Drosdowsky","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1159\/000521288","article-title":"Charlson comorbidity index: a critical review of clinimetric properties","volume":"91","author":"Charlson","journal-title":"Psychother Psychosom"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/66098","article-title":"Use of retrieval-augmented large language model for COVID-19 fact-checking: development and usability study","volume":"27","author":"Li","journal-title":"J Med Internet Res"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1002\/hast.973","article-title":"Artificial intelligence and black-box medical decisions: accuracy versus explainability","volume":"49","author":"London","journal-title":"Hastings Cent Rep"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/56306","article-title":"Finding consensus on trust in AI in health care: recommendations from a panel of international experts","volume":"27","author":"Starke","journal-title":"J Med Internet Res"},{"issue":"Suppl 1","key":"R27","doi-asserted-by":"publisher","first-page":"i119","DOI":"10.1093\/bioinformatics\/btae238","article-title":"Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models","volume":"40","author":"Jeong","journal-title":"Bioinformatics"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocn.2024.110847","article-title":"Large language models can effectively extract stroke and reperfusion audit data from medical free-text discharge summaries","volume":"129","author":"Goh","journal-title":"J Clin Neurosci"},{"issue":"3","key":"R29","doi-asserted-by":"publisher","first-page":"445","DOI":"10.3390\/medicina60030445","article-title":"Integrating retrieval-augmented generation with large language models in nephrology: advancing practical applications","volume":"60","author":"Miao","journal-title":"Med Bogota Colomb"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2025\/1\/e78393","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:11:00Z","timestamp":1782843060000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2025\/1\/e78393"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,19]]},"references-count":29,"URL":"https:\/\/doi.org\/10.2196\/78393","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,19]]},"article-number":"v27i3e78393"}}