{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T20:47:23Z","timestamp":1782506843756,"version":"3.54.5"},"reference-count":50,"publisher":"American Society of Clinical Oncology (ASCO)","issue":"2","content-domain":{"domain":["ascopubs.org"],"crossmark-restriction":true},"short-container-title":["JCO Clin Cancer Inform"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:sec>\n                    <jats:title>PURPOSE<\/jats:title>\n                    <jats:p>The rapidly evolving breast cancer treatment landscape creates significant information synthesis challenges for clinicians. We evaluated whether small open-source large language models (LLMs) augmented with retrieval-augmented generation (RAG) could match proprietary model performance for clinical guideline queries.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>METHODS<\/jats:title>\n                    <jats:p>We developed a domain-specialized RAG pipeline using HTML-structure-preserving chunking of 1,356 ASCO breast cancer guideline documents. Five LLMs were each evaluated with and without RAG: GPT-4-turbo, GPT-3.5-turbo, Qwen2.5-14B (14 billion parameters), LLaMA3-8B, and OpenBioLLM-8B. Performance was assessed using 98 expert-curated question-answer-context triplets across seven breast cancer categories. Evaluation used both rubric-based scoring (six metrics: fluency, relevance, reliability, consistency, clarity, and clinical impact) and exhaustive pairwise ranking by GPT-4-turbo as judge. Human validation was conducted with 15 practicing oncologists on a 10-query subset.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>RESULTS<\/jats:title>\n                    <jats:p>\n                      RAG-enhanced Qwen2.5-14B achieved mean rubric scores of 3.77 versus 3.96 for GPT-4-turbo and pairwise ranking performance of 0.72 versus 0.81 (normalized scale). Although absolute rubric gains were modest (0.02-0.05 on a five-point scale), relative improvements in head-to-head win rates ranged from 16% to 46%. Human expert scores confirmed RAG superiority but were consistently more conservative than LLM judge scores (mean 3.81\n                      <jats:italic toggle=\"yes\">v<\/jats:italic>\n                      4.12 across all metrics). Optimal retrieval used top-5 contexts; performance degraded sharply at higher context volumes.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>CONCLUSION<\/jats:title>\n                    <jats:p>Small open-source LLMs with optimized RAG can approach state-of-the-art proprietary model performance for clinical decision support. This approach enables scalable, cost-effective, privacy-preserving deployment without recurrent fine-tuning, suggesting potential for real-world clinical implementation on single-graphics processing unit infrastructure under expert supervision.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1200\/cci-25-00361","type":"journal-article","created":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T20:00:11Z","timestamp":1782504011000},"update-policy":"https:\/\/doi.org\/10.1200\/crossmark","source":"Crossref","is-referenced-by-count":0,"title":["Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer Clinical Decision Support?"],"prefix":"10.1200","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-7381-0943","authenticated-orcid":true,"given":"Chanhee","family":"Park","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University, Seoul, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7894-8772","authenticated-orcid":true,"given":"In Hae","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Hemato-Oncology, Korea University College of Medicine, Seoul, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minhyuk","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University, Seoul, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3591-638X","authenticated-orcid":true,"given":"Youngjoon","family":"Jang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University, Seoul, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9269-1157","authenticated-orcid":true,"given":"Heuiseok","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University, Seoul, Korea"},{"name":"Human-Inspired AI Research, Seoul, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5997-1627","authenticated-orcid":true,"given":"Yuna","family":"Hur","sequence":"additional","affiliation":[{"name":"Human-Inspired AI Research, Seoul, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"233","reference":[{"key":"e_1_3_7_2_2","doi-asserted-by":"publisher","DOI":"10.1002\/cac2.12358"},{"key":"e_1_3_7_3_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2018.19323"},{"key":"e_1_3_7_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.bj.2025.100868"},{"key":"e_1_3_7_5_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis P","year":"2020","unstructured":"Lewis P, Perez E, Piktus A, et al: Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst 33:9459-9474, 2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"e_1_3_7_6_2","doi-asserted-by":"publisher","unstructured":"Izacard G Grave E: Distilling knowledge from reader to retriever for question answering. ArXiv 10.48550\/arXiv.2012.04584","DOI":"10.48550\/arXiv.2012.04584"},{"key":"e_1_3_7_7_2","first-page":"102811","article-title":"Large language models present new questions for decision support","volume":"79","author":"Handler A","year":"2024","unstructured":"Handler A, Larsen KR, Hackathorn R: Large language models present new questions for decision support. Int J Inf Manage 79:102811, 2024","journal-title":"Int J Inf Manage"},{"key":"e_1_3_7_8_2","doi-asserted-by":"publisher","unstructured":"Chen ZZ Ma J Zhang X et al: A survey on large language models for critical societal domains: finance healthcare and law. ArXiv 10.48550\/arXiv.2405.01769","DOI":"10.48550\/arXiv.2405.01769"},{"key":"e_1_3_7_9_2","article-title":"Augmenting large language models with national comprehensive cancer network guidelines for improved and standardized adjuvant therapy recommendations in postoperative breast cancer cases","author":"Goh SSN","unstructured":"Goh SSN, Mariappan R, Soo Woon Tan G, et al: Augmenting large language models with national comprehensive cancer network guidelines for improved and standardized adjuvant therapy recommendations in postoperative breast cancer cases. JCO Clin Cancer Inform 10.1200\/CCI-24-00243","journal-title":"JCO Clin Cancer Inform"},{"key":"e_1_3_7_10_2","article-title":"Large language models as decision-making tools in oncology: Comparing artificial intelligence suggestions and expert recommendations","author":"Ah-Thiane L","unstructured":"Ah-Thiane L, Heudel PE, Campone M, et al: Large language models as decision-making tools in oncology: Comparing artificial intelligence suggestions and expert recommendations. JCO Clin Cancer Inform 10.1200\/CCI-24-00230","journal-title":"JCO Clin Cancer Inform"},{"key":"e_1_3_7_11_2","article-title":"Enhancing oncology-specific question answering with large language models through fine-tuned embeddings with synthetic data","author":"Lu KH","unstructured":"Lu KH, Mehdinia S, Man K, et al: Enhancing oncology-specific question answering with large language models through fine-tuned embeddings with synthetic data. JCO Clin Cancer Inform 10.1200\/CCI-25-00011","journal-title":"JCO Clin Cancer Inform"},{"key":"e_1_3_7_12_2","doi-asserted-by":"publisher","unstructured":"Omar M Sorin V Collins JD et al: Large language models are highly vulnerable to adversarial hallucination attacks in clinical decision support: A multi-model assurance analysis. medRxiv 10.1101\/2025.03.18.25324184","DOI":"10.1101\/2025.03.18.25324184"},{"key":"e_1_3_7_13_2","doi-asserted-by":"publisher","DOI":"10.2196\/59823"},{"key":"e_1_3_7_14_2","doi-asserted-by":"publisher","unstructured":"Howard J Ruder S: Universal language model fine-tuning for text classification. ArXiv 10.48550\/arXiv.1801.06146","DOI":"10.48550\/arXiv.1801.06146"},{"key":"e_1_3_7_15_2","doi-asserted-by":"publisher","unstructured":"Gururangan S Marasovi\u2019c A Swayamdipta S et al: Don\u2019t stop pretraining: Adapt language models to domains and tasks. ArXiv 10.48550\/arXiv.2004.10964","DOI":"10.48550\/arXiv.2004.10964"},{"key":"e_1_3_7_16_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"e_1_3_7_17_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01653-8"},{"key":"e_1_3_7_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.knowllm-1.16"},{"key":"e_1_3_7_19_2","doi-asserted-by":"publisher","unstructured":"Zhou S Liu X Xu Z et al: Mitigating ethical issues for large language models in oncology: A systematic review. JCO Clin Cancer Inform 10.1200\/CCI-25-00076","DOI":"10.1200\/CCI-25-00076"},{"key":"e_1_3_7_20_2","volume-title":"Qwen2.5: A Party of Foundation Models","author":"Team Q","year":"2024","unstructured":"Team Q: Qwen2.5: A Party of Foundation Models, 2024. https:\/\/qwenlm.github.io\/blog\/qwen2.5\/"},{"key":"e_1_3_7_21_2","doi-asserted-by":"publisher","unstructured":"OpenAI: Gpt-4 technical report. ArXiv 10.48550\/arXiv.2303.08774","DOI":"10.48550\/arXiv.2303.08774"},{"key":"e_1_3_7_22_2","doi-asserted-by":"publisher","unstructured":"Adhikari NS Agarwal S: A comparative study of pdf parsing tools across diverse document categories. ArXiv 10.48550\/arXiv.2410.09871","DOI":"10.48550\/arXiv.2410.09871"},{"key":"e_1_3_7_23_2","doi-asserted-by":"publisher","unstructured":"Singh IS Aggarwal R Allahverdiyev I et al: Chunkrag: Novel LLM-chunk filtering method for rag systems. ArXiv 10.48550\/arXiv.2410.19572","DOI":"10.48550\/arXiv.2410.19572"},{"key":"e_1_3_7_24_2","doi-asserted-by":"publisher","unstructured":"Bhat SR Rudat M Spiekermann J et al: Rethinking chunk size for long-document retrieval: A multi-dataset analysis. ArXiv 10.48550\/arXiv.2505.21700","DOI":"10.48550\/arXiv.2505.21700"},{"key":"e_1_3_7_25_2","doi-asserted-by":"publisher","DOI":"10.1177\/20552076251337177"},{"key":"e_1_3_7_26_2","doi-asserted-by":"publisher","unstructured":"Brown TB Mann B Ryder N et al: Language models are few-shot learners. In: NeurIPS. ArXiv 10.48550\/arXiv.2005.14165","DOI":"10.48550\/arXiv.2005.14165"},{"key":"e_1_3_7_27_2","doi-asserted-by":"publisher","unstructured":"Grattafiori A Dubey A Jauhri A et al: The Llama 3 herd of model. ArXiv 10.48550\/arXiv.2407.21783","DOI":"10.48550\/arXiv.2407.21783"},{"key":"e_1_3_7_28_2","volume-title":"OpenBioLLMs: Advancing open-source large language models for healthcare and life sciences. Hugging Face model card","author":"Pal A","year":"2024","unstructured":"Pal A, Sankarasubbu M: OpenBioLLMs: Advancing open-source large language models for healthcare and life sciences. Hugging Face model card, 2024. https:\/\/huggingface.co\/aaditya\/Llama3-OpenBioLLM-8B"},{"key":"e_1_3_7_29_2","doi-asserted-by":"publisher","DOI":"10.2196\/53216"},{"key":"e_1_3_7_30_2","doi-asserted-by":"publisher","unstructured":"Sclar M Choi Y Tsvetkov Y et al: Quantifying language models\u2019 sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. ArXiv 10.48550\/arXiv.2310.11324","DOI":"10.48550\/arXiv.2310.11324"},{"key":"e_1_3_7_31_2","doi-asserted-by":"publisher","DOI":"10.3389\/fbinf.2024.1457619"},{"key":"e_1_3_7_32_2","doi-asserted-by":"publisher","unstructured":"Deutsch D Dror R Roth D: On the limitations of reference-free evaluations of generated text. ArXiv 10.48550\/arXiv.2210.12563","DOI":"10.48550\/arXiv.2210.12563"},{"key":"e_1_3_7_33_2","doi-asserted-by":"publisher","unstructured":"Blagec K Dorffner G Moradi M et al: A global analysis of metrics used for measuring performance in natural language processing. ArXiv 10.48550\/arXiv.2204.11574","DOI":"10.48550\/arXiv.2204.11574"},{"key":"e_1_3_7_34_2","doi-asserted-by":"publisher","unstructured":"Zhang T Kishore V Wu F et al: Bertscore: Evaluating text generation with bert. ArXiv 10.48550\/arXiv.1904.09675","DOI":"10.48550\/arXiv.1904.09675"},{"key":"e_1_3_7_35_2","doi-asserted-by":"publisher","unstructured":"Gu J Jiang X Shi Z et al: A survey on LLM-as-a-judge. ArXiv 10.48550\/arXiv.2411.15594","DOI":"10.48550\/arXiv.2411.15594"},{"key":"e_1_3_7_36_2","doi-asserted-by":"publisher","unstructured":"Wang Y Hernandez AG Kyslyi R et al: Evaluating quality of answers for retrieval-augmented generation: A strong LLM is all you need. ArXiv 10.48550\/arXiv.2406.18064","DOI":"10.48550\/arXiv.2406.18064"},{"key":"e_1_3_7_37_2","doi-asserted-by":"crossref","unstructured":"Wang R Guo J Gao C et al Can LLMs replace human evaluators? An empirical study of LLM-as-a-judge in software engineering. Proceedings of the ACM on Software Engineering 2 (ISSTA)","DOI":"10.1145\/3728963"},{"key":"e_1_3_7_38_2","doi-asserted-by":"publisher","unstructured":"Alwakeel M Nagori A Krishnamoorthy V et al: Evaluating LLMs in medicine: A call for rigor transparency. ArXiv 10.48550\/arXiv.2507.08916","DOI":"10.48550\/arXiv.2507.08916"},{"key":"e_1_3_7_39_2","doi-asserted-by":"publisher","unstructured":"Ngo NT Van Nguyen C Dernoncourt F et al: Comprehensive and practical evaluation of retrieval-augmented generation systems for medical question answering. ArXiv 10.48550\/arXiv.2411.09213","DOI":"10.48550\/arXiv.2411.09213"},{"key":"e_1_3_7_40_2","doi-asserted-by":"publisher","unstructured":"Wu K Wu E Cassasola A et al: How well do LLMs cite relevant medical references? An evaluation framework and analyses. ArXiv 10.48550\/arXiv.2402.02008","DOI":"10.48550\/arXiv.2402.02008"},{"key":"e_1_3_7_41_2","doi-asserted-by":"publisher","unstructured":"Shi L Ma C Liang W et al: Judging the judges: A systematic study of position bias in LLM-as-a-Judge. ArXiv 10.48550\/arXiv.2406.07791","DOI":"10.48550\/arXiv.2406.07791"},{"key":"e_1_3_7_42_2","doi-asserted-by":"publisher","unstructured":"Liusie A Manakul P Gales MJF: LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. ArXiv 10.48550\/arXiv.2307.07889","DOI":"10.48550\/arXiv.2307.07889"},{"key":"e_1_3_7_43_2","doi-asserted-by":"publisher","unstructured":"Liu A Feng B Xue B et al: DeepSeek-V3 technical report. ArXiv 10.48550\/arXiv.2412.19437","DOI":"10.48550\/arXiv.2412.19437"},{"key":"e_1_3_7_44_2","doi-asserted-by":"publisher","unstructured":"Bi B Liu S Wang Y et al: Parameters vs. context: Fine-grained control of knowledge reliance in language models. ArXiv 10.48550\/arXiv.2503.15888","DOI":"10.48550\/arXiv.2503.15888"},{"key":"e_1_3_7_45_2","doi-asserted-by":"publisher","unstructured":"Dorfner FJ Dada A Busch F et al: Biomedical large language models seem not to be superior to generalist models on unseen medical data. ArXiv 10.48550\/arXiv.2408.13833","DOI":"10.48550\/arXiv.2408.13833"},{"key":"e_1_3_7_46_2","doi-asserted-by":"publisher","unstructured":"Fu X-Y Laskar MTR Khasanova E et al: Tiny titans: Can smaller large language models punch above their weight in the real world for meeting summarization? ArXiv 10.48550\/arXiv.2402.00841","DOI":"10.48550\/arXiv.2402.00841"},{"key":"e_1_3_7_47_2","doi-asserted-by":"publisher","unstructured":"Pecher B Srba I Bielikova M: Comparing specialised small and general large language models on text classification: 100 labelled samples to achieve break-even performance. ArXiv 10.48550\/arXiv.2402.12819","DOI":"10.48550\/arXiv.2402.12819"},{"key":"e_1_3_7_48_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-emnlp.41"},{"key":"e_1_3_7_49_2","article-title":"Use of large language models in clinical cancer research","author":"Kehl KL","unstructured":"Kehl KL: Use of large language models in clinical cancer research. JCO Clin Cancer Inform 10.1200\/CCI-25-00027","journal-title":"JCO Clin Cancer Inform"},{"key":"e_1_3_7_50_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638"},{"key":"e_1_3_7_51_2","article-title":"Effect of an artificial intelligence clinical decision support system on treatment decisions for complex breast cancer","author":"Xu F","unstructured":"Xu F, Sep\u00falveda MJ, Jiang Z, et al: Effect of an artificial intelligence clinical decision support system on treatment decisions for complex breast cancer. JCO Clin Cancer Inform 10.1200\/CCI.20.00018","journal-title":"JCO Clin Cancer Inform"}],"container-title":["JCO Clinical Cancer Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ascopubs.org\/doi\/pdfdirect\/10.1200\/CCI-25-00361","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T20:00:23Z","timestamp":1782504023000},"score":1,"resource":{"primary":{"URL":"https:\/\/ascopubs.org\/doi\/10.1200\/CCI-25-00361"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4]]},"references-count":50,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["10.1200\/CCI-25-00361"],"URL":"https:\/\/doi.org\/10.1200\/cci-25-00361","relation":{},"ISSN":["2473-4276"],"issn-type":[{"value":"2473-4276","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4]]},"assertion":[{"value":"2025-11-24","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-26","order":1,"name":"revised","label":"Revised","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2500361"}}