{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T16:28:39Z","timestamp":1787243319768,"version":"build-2736575974"},"reference-count":38,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Maintenance of oncology clinical practice guidelines (CPGs) is increasingly challenged by the rapid growth of trial data and therapeutic complexity. While large language models (LLMs) have shown promise in information retrieval, their utility in the rigorous, end-to-end workflow of guideline maintenance remains underexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This case study aimed to systematically evaluate the performance of frontier LLMs in supporting oncology guideline maintenance. We sought to determine their reliability in predicting necessary guideline updates based on new evidence, their accuracy in extracting data from clinical trials, and their effectiveness as automated auditors for detecting errors in established guidelines.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Using the Onkopedia peripheral T-cell lymphoma (PTCL) guideline as a prospective case study, we tasked frontier models with deep-research modes and autonomous web-search capabilities (Gemini 2.5 Pro and GPT o4-mini-high) to predict a guideline update in August 2025 based on the 2021 version. Predictions were validated against the official 2025 revision published in October 2025. Next, we benchmarked evidence extraction accuracy across 80 pivotal trials using models of varying scale (27B-671B parameters vs frontier). Finally, we deployed a stacked LLM workflow to audit 28 recently updated Onkopedia guidelines for linguistic and content-related errors.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n In the predictive task, models captured 36.7% to 40% of substantive updates, often identifying landmark approvals, but frequently overstating evidence. An independent, model-blinded rescoring yielded substantial agreement (weighted Cohen \u03ba=0.75) and confirmed predictive accuracies of 35% to 38.3%. While frontier models demonstrated high accuracy (up to 99.2%) in extracting data from individual studies, substantially outperforming smaller open-source models, this precision declined during multisource synthesis. We observed a position-dependent performance drop in long-form generation, with GPT\u2019s endpoint accuracy dropping from 84.6% in the first half of the drafted guideline to 38.5% in the second half. As automated auditors of existing CPGs, the models successfully identified a median of 16.5 (IQR 13.8-20.3) formal errors per document and detected several clinically relevant inconsistencies (eg, invalid scoring formulas and incorrect staging definitions).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLMs currently lack the reasoning stability for autonomous guideline authoring due to deficits in complex synthesis. However, they are effective tools for high-fidelity evidence extraction and automated quality assurance, supporting a human-led, AI-augmented workflow for efficient guideline maintenance.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/93239","type":"journal-article","created":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T12:40:06Z","timestamp":1783600806000},"page":"e93239-e93239","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models for Oncology Guideline Maintenance: Prospective Case Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3475-2396","authenticated-orcid":false,"given":"Manuel","family":"Knauer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6153-3256","authenticated-orcid":false,"given":"Julian","family":"Gre\u00df","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3730-5348","authenticated-orcid":false,"given":"Jakob Nikolas","family":"Kather","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9045-0595","authenticated-orcid":false,"given":"Peter","family":"May","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,20]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01810-z","article-title":"Large language models-enabled digital twins for precision medicine in rare gynecological tumors","volume":"8","author":"Lammert","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01233-2","article-title":"Privacy-preserving large language models for structured medical information retrieval","volume":"7","author":"Wiest","journal-title":"NPJ Digit Med"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/78332","article-title":"Enabling just-in-time clinical oncology analysis with large language models: feasibility and validation study using unstructured synthetic data","volume":"13","author":"May","journal-title":"JMIR Med Inform"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1038\/s41698-024-00733-4","article-title":"Large language model use in clinical oncology","volume":"8","author":"Carl","journal-title":"NPJ Precis Oncol"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01824-7","article-title":"Large language model integrations in cancer decision-making: a systematic review and meta-analysis","volume":"8","author":"Hao","journal-title":"NPJ Digit Med"},{"issue":"4","key":"R6","doi-asserted-by":"publisher","DOI":"10.3390\/biomedicines13040951","article-title":"Artificial intelligence advancements in oncology: a review of current trends and future directions","volume":"13","author":"Huhulea","journal-title":"Biomedicines"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.2196\/65984","article-title":"Large language model applications for health information extraction in oncology: scoping review","volume":"11","author":"Chen","journal-title":"JMIR Cancer"},{"issue":"12","key":"R8","doi-asserted-by":"publisher","first-page":"1447","DOI":"10.1016\/j.annonc.2025.09.001","article-title":"ESMO guidance on the use of large language models in clinical practice (ELCAP)","volume":"36","author":"Wong","journal-title":"Ann Oncol"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1093\/intqhc\/mzv115","article-title":"Guide to clinical practice guidelines: the current state of play","volume":"28","author":"Kredo","journal-title":"Int J Qual Health Care"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-022-08975-3","article-title":"Approaches to clinical guideline development in healthcare: a scoping review and document analysis","volume":"23","author":"De Leo","journal-title":"BMC Health Serv Res"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1016\/j.injury.2022.01.047","article-title":"Clinical practice guidelines: the good, the bad, and the ugly","volume":"54 Suppl 3","author":"Guerra-Farfan","journal-title":"Injury"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1186\/1748-5908-6-107","article-title":"The updating of clinical practice guidelines: insights from an international survey","volume":"6","author":"Alonso-Coello","journal-title":"Implement Sci"},{"issue":"3","key":"R13","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.0841","article-title":"Changes in length and complexity of clinical practice guidelines in oncology, 1996-2019","volume":"3","author":"Kann","journal-title":"JAMA Netw Open"},{"issue":"16","key":"R14","doi-asserted-by":"publisher","first-page":"1211","DOI":"10.1503\/cmaj.140547","article-title":"The validity of recommendations from clinical guidelines: a survival analysis","volume":"186","author":"Mart\u00ednez Garc\u00eda","journal-title":"CMAJ"},{"issue":"8","key":"R15","doi-asserted-by":"publisher","first-page":"1154","DOI":"10.7326\/M22-0514","article-title":"A framework for the development of living practice guidelines in health care","volume":"175","author":"El Mikati","journal-title":"Ann Intern Med"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2022.12.022","article-title":"Methods for living guidelines: early guidance based on practical experience. Paper 5: decisions on methods for evidence synthesis and recommendation development for living guidelines","volume":"155","author":"Fraile Navarro","journal-title":"J Clin Epidemiol"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.11.037","article-title":"A pilot living clinical practice guidelines approach was feasible and acceptable to guideline panel members","volume":"143","author":"Bragge","journal-title":"J Clin Epidemiol"},{"issue":"10","key":"R18","doi-asserted-by":"publisher","DOI":"10.1093\/oncolo\/oyaf293","article-title":"Large language model processing capabilities of ChatGPT 4.0 to generate molecular tumor board recommendations-a critical evaluation on real world data","volume":"30","author":"Schmutz","journal-title":"Oncologist"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01840-7","article-title":"Accelerating clinical evidence synthesis with large language models","volume":"8","author":"Wang","journal-title":"NPJ Digit Med"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/67488","article-title":"Accuracy of large language models for literature screening in thoracic surgery: diagnostic study","volume":"27","author":"Dai","journal-title":"J Med Internet Res"},{"issue":"9","key":"R21","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","journal-title":"Nat Med"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1200\/CCI-24-00243","article-title":"Augmenting large language models with National Comprehensive Cancer Network guidelines for improved and standardized adjuvant therapy recommendations in postoperative breast cancer cases","volume":"9","author":"Goh","journal-title":"JCO Clin Cancer Inform"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01091-y","article-title":"Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework","volume":"7","author":"Kresevic","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s00062-025-01562-z","article-title":"Evaluating retrieval augmented generation-enhanced large language models for question answering on German neurovascular guidelines","volume":"36","author":"Vach","journal-title":"Clin Neuroradiol"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.acl-long.729","article-title":"GraphCheck: breaking long-term text barriers with extracted knowledge graph-powered fact-checking","volume":"2025","author":"Chen","journal-title":"Proc Conf Assoc Comput Linguist Meet"},{"issue":"6","key":"R26","doi-asserted-by":"publisher","first-page":"3539","DOI":"10.1007\/s00464-025-11723-3","article-title":"Large language model-generated clinical practice guideline for appendicitis","volume":"39","author":"Boyle","journal-title":"Surg Endosc"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1200\/CCI-25-00194","article-title":"Artificial intelligence-assisted error detection in complex clinical documentation: leveraging large language models to enhance patient safety in oncology","volume":"10","author":"May","journal-title":"JCO Clin Cancer Inform"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: how language models use long contexts","volume":"12","author":"Liu","journal-title":"Trans Assoc Comput Linguist"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/53164","article-title":"Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews: comparative analysis","volume":"26","author":"Chelli","journal-title":"J Med Internet Res"},{"issue":"7","key":"R30","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.20496","article-title":"Performance of a large language model in screening citations","volume":"7","author":"Oami","journal-title":"JAMA Netw Open"},{"issue":"4","key":"R31","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1002\/jrsm.1715","article-title":"Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages","volume":"15","author":"Khraisha","journal-title":"Res Synth Methods"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.2196\/48996","article-title":"Automated paper screening for clinical reviews using large language models: data analysis study","volume":"26","author":"Guo","journal-title":"J Med Internet Res"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250501","article-title":"Large language model-assisted systematic review: validation based on Cochrane review data","volume":"327","author":"Kim","journal-title":"Stud Health Technol Inform"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2022.08.013","article-title":"In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving","volume":"153","author":"Marshall","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"R35","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.242575","article-title":"Generative large language models trained for detecting errors in radiology reports","volume":"315","author":"Sun","journal-title":"Radiology"},{"issue":"13","key":"R36","doi-asserted-by":"publisher","first-page":"1231","DOI":"10.1182\/blood.2023022204","article-title":"Antitumor efficacy and safety of unedited autologous CD5.CAR T cells in relapsed\/refractory mature T-cell lymphomas","volume":"143","author":"Hill","journal-title":"Blood"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.2196\/92642","article-title":"Sequencing AI automation and data interoperability in oncology using a scenario-planning framework coupled with discrete-event simulation: proof-of-concept study","volume":"28","author":"May","journal-title":"J Med Internet Res"},{"key":"R38","unstructured":"May P . Augmenting oncology guideline maintenance with large language models: a prospective case study (JMIR AI). Zenodo. 2026. URL: https:\/\/zenodo.org\/records\/21219226 [Accessed 03-08-2026]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T16:13:52Z","timestamp":1787242432000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e93239"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,20]]},"references-count":38,"URL":"https:\/\/doi.org\/10.2196\/93239","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,20]]},"article-number":"v5i1e93239"}}