{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T17:43:11Z","timestamp":1790962991919,"version":"4.1.0"},"posted":{"date-parts":[[2025,8,11]]},"group-title":"Oncology","reference-count":26,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,8,11]],"date-time":"2025-08-11T00:00:00Z","timestamp":1754870400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,8,11]]},"abstract":"<jats:p>Effective risk communication is essential to shared decision-making in prostate cancer care. However, the quality of physician communication of key tradeoffs varies widely in real-world consultations. Manual evaluation of communication is labor-intensive and not scalable. We present a structured, rubric-based framework that uses large language models (LLMs) to automatically score the quality of risk communication in prostate cancer consultations. Using transcripts from 20 clinical visits, we curated and annotated 487 physician-spoken sentences that referenced five decision-making domains: cancer prognosis, life expectancy, and three treatment side effects (erectile dysfunction, incontinence, and irritative urinary symptoms). Each sentence was assigned a score from 0 to 5 based on the precision and patient-specificity of communicated risk, using a validated scoring rubric. We modeled this task as five multiclass classification problems and evaluated both fine-tuned transformer baselines and GPT-4o with rubric-based and chain-of-thought (CoT) prompting. Our best performing approach, which combined rubric-based CoT prompting with few-shot learning, achieved micro averaged F1 scores between 85.0 and 92.0 across domains, outperforming supervised baselines and matching inter-annotator agreement. These findings establish a scalable foundation for AI-driven evaluation of physician\u2013patient communication in oncology and beyond.<\/jats:p>","DOI":"10.1101\/2025.08.07.25333034","type":"posted-content","created":{"date-parts":[[2025,8,12]],"date-time":"2025-08-12T01:35:25Z","timestamp":1754962525000},"source":"Crossref","is-referenced-by-count":1,"title":["Scoring Physician Risk Communication in Prostate Cancer Using Large Language Models"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5903-1483","authenticated-orcid":false,"given":"Guillermo","family":"Lopez-Garcia","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0828-1102","authenticated-orcid":false,"given":"Dongfang","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Luu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renning","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timothy J.","family":"Daskivich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Graciela","family":"Gonzalez-Hernandez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2025081301500902000_2025.08.07.25333034v1.1","doi-asserted-by":"crossref","first-page":"370","DOI":"10.7326\/0003-4819-159-5-201309030-00018","article-title":"Effect of Age, Tumor Risk, and Comorbidity in a U.S. Population\u2013Based Cohort of Men With Prostate Cancer","volume":"159","year":"2013","journal-title":"Annals of Internal Medicine"},{"key":"2025081301500902000_2025.08.07.25333034v1.2","doi-asserted-by":"publisher","DOI":"10.1007\/s11136-020-02503-8"},{"key":"2025081301500902000_2025.08.07.25333034v1.3","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.7248"},{"key":"2025081301500902000_2025.08.07.25333034v1.4","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1016\/j.eururo.2020.02.009","article-title":"Radical Prostatectomy or Observation for Clinically Localized Prostate Cancer: Extended Follow-up of the Prostate Cancer Intervention Versus Observation Trial (PIVOT)","volume":"77","year":"2020","journal-title":"European Urology"},{"key":"2025081301500902000_2025.08.07.25333034v1.5","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1016\/j.urpr.2015.10.006","article-title":"AUA White Paper on Implementation of Shared Decision Making into Urological Practice","volume":"3","year":"2016","journal-title":"Urology Practice"},{"key":"2025081301500902000_2025.08.07.25333034v1.6","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2010.31.2330"},{"key":"2025081301500902000_2025.08.07.25333034v1.7","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1001\/jamasurg.2021.5803","article-title":"Patient Preferences for Communication of Life Expectancy in Prostate Cancer Treatment Con-sultations","volume":"157","year":"2022","journal-title":"JAMA Surgery"},{"key":"2025081301500902000_2025.08.07.25333034v1.8","doi-asserted-by":"crossref","first-page":"2319","DOI":"10.1056\/NEJMoa1807801","article-title":"Radical Prostatectomy or Watchful Waiting in Prostate Cancer \u2014 29-Year Follow-up","volume":"379","year":"2018","journal-title":"New England Journal of Medicine"},{"key":"2025081301500902000_2025.08.07.25333034v1.9","doi-asserted-by":"crossref","first-page":"e25659","DOI":"10.2196\/25659","article-title":"Exploring Cancer Survivor Needs and Preferences for Communicating Personalized Cancer Statistics From Registry Data: Qualitative Multimethod Study","volume":"7","year":"2021","journal-title":"JMIR Cancer"},{"key":"2025081301500902000_2025.08.07.25333034v1.10","doi-asserted-by":"crossref","unstructured":"G. Elwyn , P. J. Barr and S. W. Grande , Patients recording clinical encounters: a path to empowerment? assessment by mixed methods, BMJ Open 5 (2015).","DOI":"10.1136\/bmjopen-2015-008566"},{"key":"2025081301500902000_2025.08.07.25333034v1.11","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1097\/JU.0000000000002675","article-title":"Variation in Communication of Competing Risks of Mortality in Prostate Cancer Treatment Consultations","volume":"208","year":"2022","journal-title":"The Journal of Urology"},{"key":"2025081301500902000_2025.08.07.25333034v1.12","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1038\/s41391-024-00806-2","article-title":"Variation in communication of side effects in prostate cancer treatment consultations","volume":"28","year":"2025","journal-title":"Prostate Cancer and Prostatic Diseases"},{"key":"2025081301500902000_2025.08.07.25333034v1.13","first-page":"e392","article-title":"MP24-03 Development and Validation of a Natural Language Processing System to Identify Information on Key Tradeoffs in Prostate Cancer Treatment Consultations","volume":"211","year":"2024","journal-title":"Journal of Urology"},{"key":"2025081301500902000_2025.08.07.25333034v1.14","first-page":"e798","article-title":"MP23-09 A Framework for Evaluating the Quality of Risk Communication in Prostate Cancer Treatment Consultations","volume":"213","year":"2025","journal-title":"Journal of Urology"},{"key":"2025081301500902000_2025.08.07.25333034v1.15","doi-asserted-by":"crossref","unstructured":"A. Jerfy , O. Selden and R. Balkrishnan , The Growing Impact of Natural Language Processing in Healthcare and Public Health, INQUIRY: The Journal of Health Care Organization, Provision, and Financing 61 (2024).","DOI":"10.1177\/00469580241290095"},{"key":"2025081301500902000_2025.08.07.25333034v1.16","doi-asserted-by":"crossref","unstructured":"M. Cascella , F. Semeraro , J. Montomoli , V. Bellini , O. Piazza and E. Bignami , The Breakthrough of Large Language Models Release for Medical Applications: 1-Year Timeline and Perspectives, Journal of Medical Systems 48 (2024).","DOI":"10.1007\/s10916-024-02045-3"},{"key":"2025081301500902000_2025.08.07.25333034v1.17","unstructured":"R. Zheng , N. A. Friedrich , M. Luu , R. Gale , D. Khodyakov , S. J. Freedland , B. Spiegel and T. J. Daskivich , Development and Validation of a Natural Language Processing System to Assess Quality of Physician Communication in Prostate Cancer Consultations, Prostate Cancer and Prostatic Diseases (2025)."},{"key":"2025081301500902000_2025.08.07.25333034v1.18","unstructured":"J. Wei , X. Wang , D. Schuurmans , M. Bosma , b. ichter , F. Xia , E. Chi , Q.V. Le and D. Zhou , Chain-of-thought prompting elicits reasoning in large language models, in Advances in Neural Information Processing Systems, eds. S. Koyejo , S. Mohamed , A. Agarwal , D. Belgrave , K. Cho and A. Oh (Curran Associates, Inc., 2022)."},{"key":"2025081301500902000_2025.08.07.25333034v1.19","unstructured":"T. Brown , B. Mann , N. Ryder , M. Subbiah , J. D. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger , T. Henighan , R. Child , A. Ramesh , D. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler , M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever and D. Amodei , Language Models are Few-Shot Learners, in Advances in Neural Information Processing Systems, eds. H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan and H. Lin (Curran Associates, Inc., 2020)."},{"key":"2025081301500902000_2025.08.07.25333034v1.20","first-page":"276","article-title":"Interrater reliability: the kappa statistic","volume":"22","year":"2012","journal-title":"Biochemia Medica"},{"key":"2025081301500902000_2025.08.07.25333034v1.21","unstructured":"T. Kojima , S. S. Gu , M. Reid , Y. Matsuo and Y. Iwasawa , Large language models are zero-shot reasoners, in Advances in Neural Information Processing Systems, eds. S. Koyejo , S. Mohamed , Agarwal, D. Belgrave , K. Cho and A. Oh (Curran Associates, Inc., 2022)."},{"key":"2025081301500902000_2025.08.07.25333034v1.22","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104458"},{"key":"2025081301500902000_2025.08.07.25333034v1.23","unstructured":"J. Devlin , M.-W. Chang , K. Lee and K. Toutanova , BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), (Association for Computational Linguistics, Minneapolis, Minnesota, June 2019)."},{"key":"2025081301500902000_2025.08.07.25333034v1.24","unstructured":"A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , L. u. Kaiser and I. Polosukhin , Attention is All you Need, in Advances in Neural Information Processing Systems, eds. I. Guyon , U. V. Luxburg , S. Bengio , H. Wallach , R. Fergus , S. Vishwanathan and R. Garnett (Curran Associates, Inc., 2017)."},{"key":"2025081301500902000_2025.08.07.25333034v1.25","unstructured":"Y. Liu , M. Ott , N. Goyal , J. Du , M. Joshi , D. Chen , O. Levy , M. Lewis , L. Zettlemoyer and V. Stoyanov , RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019)."},{"key":"2025081301500902000_2025.08.07.25333034v1.26","doi-asserted-by":"crossref","first-page":"100729","DOI":"10.1016\/j.patter.2023.100729","article-title":"Fine-tuning large neural language models for biomedical natural language processing","volume":"4","year":"2023","journal-title":"Patterns"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.08.07.25333034","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:42:46Z","timestamp":1768509766000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2025.08.07.25333034"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,11]]},"references-count":26,"URL":"https:\/\/doi.org\/10.1101\/2025.08.07.25333034","relation":{},"subject":[],"published":{"date-parts":[[2025,8,11]]},"subtype":"preprint"}}