{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T13:49:06Z","timestamp":1790689746549,"version":"4.1.0"},"reference-count":18,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) can accurately classify biomedical documents, but strong benchmark performance does not establish that predictions are grounded in the supplied text. In biomedical literature tasks, titles, abstracts, digital object identifiers (DOIs), journal metadata, and trial identifiers may have been seen during pretraining and can trigger parametric knowledge or learned associations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to test whether oncology randomized trial success classification is driven by abstract evidence or by identifier-conditioned prior knowledge, and assess whether models follow counterfactual outcome evidence when it conflicts with original trial identifiers.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We evaluated 250 two-arm oncology randomized controlled trials from 7 major journals published between 2005 and 2023, each with a single primary endpoint and previously adjudicated positive or negative ground-truth label. The corpus included 58.4% (146\/250) positive and 41.6% (104\/250) negative trials. GPT-5.2, Gemini 3 Flash, and Claude Opus 4.5 were queried via vendor APIs under default settings using a single-token output instruction. For each trial, we created 5 deterministic input conditions: title+abstract, title only, DOI only, counterfactual title+abstract in which the primary endpoint outcome statement was minimally flipped, and the same counterfactual input paired with the original DOI to create an identifier-text conflict. Performance was assessed using valid format rate, accuracy, sensitivity, specificity, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The models showed high format adherence, with valid prediction rates of 97.2% to 100%. In the title+abstract condition, all models achieved high and balanced performance (accuracy and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.96-0.97; sensitivity=0.96-0.97; specificity=0.96-0.98). Removing evidence reduced performance stepwise: title-only accuracy and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score fell to 0.79 to 0.88, and DOI-only performance fell to 0.63-0.67, exceeding the 58.4% majority class baseline but indicating limited identifier-driven signal. Counterfactual edits were concentrated in outcome-bearing text, with the Results and Conclusions sections modified for all trials, whereas the titles and Methods sections required edits in only 5.2% (13\/250) and 1.6% (4\/250) of trials. Against inverted labels, models followed counterfactual evidence with near-ceiling performance (accuracy and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.96-0.99). Reintroducing the original DOI caused little change for GPT-5.2 (accuracy and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.99) but modestly reduced\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores for Gemini (0.97) and Claude (0.95), mainly through lower sensitivity.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The evaluated LLMs robustly followed explicit end point statements in abstracts, including when those statements contradicted original trial outcomes. However, above-chance title-only and DOI-only performance, together with small decrements under counterfactual DOI conflicts, showed that identifiers can carry predictive signal and occasionally compete with textual evidence. Progressive content removal combined with counterfactual identifier-text conflicts offers a practical, reproducible audit for grounding in biomedical LLM evaluations.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/95565","type":"journal-article","created":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T19:21:32Z","timestamp":1785352892000},"page":"e95565-e95565","source":"Crossref","is-referenced-by-count":1,"title":["Evidence Use and Identifier-Conditioned Prior Knowledge in Large Language Model Classification of Oncology Trials Assessed Through Progressive Content Removal and Counterfactual Testing: Comparative Analysis"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1040-4888","authenticated-orcid":false,"given":"Paul","family":"Windisch","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9448-0786","authenticated-orcid":false,"given":"Carole","family":"Koechli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5374-8720","authenticated-orcid":false,"given":"Fabio","family":"Dennst\u00e4dt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9493-3834","authenticated-orcid":false,"given":"Daniel M","family":"Aebersold","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4359-7735","authenticated-orcid":false,"given":"Daniel R","family":"Zwahlen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7664-9207","authenticated-orcid":false,"given":"Robert","family":"F\u00f6rster","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4168-2795","authenticated-orcid":false,"given":"Christina","family":"Schr\u00f6der","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,29]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-53081-z","article-title":"Matching patients to clinical trials with large language models","volume":"15","author":"Jin","journal-title":"Nat Commun"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00970-0","article-title":"Large language models to identify social determinants of health in electronic health records","volume":"7","author":"Guevara","journal-title":"NPJ Digit Med"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2024.105531","article-title":"Implementation and evaluation of an additional GPT-4-based reviewer in PRISMA-based medical systematic literature reviews","volume":"189","author":"Landschaft","journal-title":"Int J Med Inform"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Petroni F Rockt\u00e4schel T Riedel S et al. Language models as knowledge bases? In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics; 2019:2463-2473. [doi: 10.18653\/v1\/D19-1250]","DOI":"10.18653\/v1\/D19-1250"},{"issue":"7972","key":"R5","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"key":"R6","unstructured":"Carlini N Tramer F Wallace E et al. Extracting training data from large language models. arXiv. Preprint posted online on Dec 14, 2020. [doi: 10.48550\/arXiv.2012.07805]"},{"key":"R7","doi-asserted-by":"crossref","unstructured":"Hayes J Swanberg M Chaudhari H et al. Measuring memorization in language models via probabilistic extraction. In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2025:9266-9291. [doi: 10.18653\/v1\/2025.naacl-long.469]","DOI":"10.18653\/v1\/2025.naacl-long.469"},{"key":"R8","doi-asserted-by":"crossref","unstructured":"Li Y Guo Y Guerin F Lin C . An open-source data contamination report for large language models. In: Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics; 2024:528-541. [doi: 10.18653\/v1\/2024.findings-emnlp.30]","DOI":"10.18653\/v1\/2024.findings-emnlp.30"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Du K Sn\u00e6bjarnarson V Stoehr N White J Schein A Cotterell R . Context versus prior knowledge in language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2024:13211-13235. [doi: 10.18653\/v1\/2024.acl-long.714]","DOI":"10.18653\/v1\/2024.acl-long.714"},{"key":"R10","doi-asserted-by":"crossref","unstructured":"Zhao Z Monti E Lehmann J Assem H . Enhancing contextual understanding in large language models through contrastive decoding. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2024:4225-4237. [doi: 10.18653\/v1\/2024.naacl-long.237]","DOI":"10.18653\/v1\/2024.naacl-long.237"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Neeman E Aharoni R Honovich O Choshen L Szpektor I Abend O . DisentQA: disentangling parametric and contextual knowledge with counterfactual question answering. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2023:10056-10070. [doi: 10.18653\/v1\/2023.acl-long.559]","DOI":"10.18653\/v1\/2023.acl-long.559"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02336-0","article-title":"HealthContradict: evaluating biomedical knowledge conflicts in language models","volume":"9","author":"Zhang","journal-title":"NPJ Digit Med"},{"issue":"8","key":"R13","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.26021","article-title":"Fidelity of medical reasoning in large language models","volume":"8","author":"Bedi","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01021-3","article-title":"Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support","volume":"5","author":"Omar","journal-title":"Commun Med (Lond)"},{"issue":"9","key":"R15","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and adoption of large language models in medicine","volume":"330","author":"Shah","journal-title":"JAMA"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.2196\/78221","article-title":"Large language models for supporting clear writing and detecting spin in randomized controlled trials in oncology: comparative analysis of GPT models and prompts","volume":"12","author":"Koechli","journal-title":"JMIR Cancer"},{"key":"R17","unstructured":"Regulation of human research in Switzerland. Federal Office of Public Health. URL: https:\/\/www.bag.admin.ch\/en\/regulation-of-human-research-in-switzerland [Accessed 03-07-2026]"},{"key":"R18","unstructured":"windisch-paul\/llm_memory. GitHub. URL: https:\/\/github.com\/windisch-paul\/llm_memory [Accessed 17-07-2026]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e95565","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:23:20Z","timestamp":1785536600000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e95565"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,29]]},"references-count":18,"URL":"https:\/\/doi.org\/10.2196\/95565","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,29]]},"article-number":"v5i1e95565"}}