{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T00:10:30Z","timestamp":1783642230359,"version":"3.55.0"},"reference-count":32,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Traditional cancer registries, limited by labor-intensive manual data abstraction and rigid, predefined schemas, often hinder timely and comprehensive oncology research. While large language models (LLMs) have shown promise in automating data extraction, their potential to perform direct, just-in-time (JIT) analysis on unstructured clinical narratives\u2014potentially bypassing intermediate structured databases for many analytical tasks\u2014remains largely unexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate whether a state-of-the-art LLM (Gemini 2.5 Pro) can enable a JIT clinical oncology analysis paradigm by assessing its ability to (1) perform high-fidelity multiparameter data extraction, (2) answer complex clinical queries directly from raw text, (3) automate multistep survival analyses including executable code generation, and (4) generate novel, clinically plausible hypotheses from free-text documentation.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n A synthetic dataset of 240 unstructured clinical letters from patients with stage IV non\u2013small cell lung cancer (NSCLC), embedding 14 predefined variables, was used. Gemini 2.5 Pro was evaluated on four core JIT capabilities. Performance was measured by using the following metrics: extraction accuracy (compared to human extraction of n=40 letters and across the full n=240 dataset); numerical deviation for direct question answering (n=40 to 240 letters, 5 questions); log-rank\n P<\/jats:italic>\n value and Harrell concordance index for LLM-generated versus ground-truth Kaplan-Meier survival analyses (n=160 letters, overall survival and progression-free survival); and correct justification, novelty, and a qualitative evaluation of LLM-generated hypotheses (n=80 and n=160 letters).\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n For multiparameter extraction from 40 letters, the LLM achieved >99% average accuracy, comparable to human extraction, but in significantly less time (LLM: 3.7 min vs human: 133.8 min). Across the full 240-letter dataset, LLM multiparameter extraction maintained >98% accuracy for most variables. The LLM answered multiconditional clinical queries directly from raw text with a relative deviation rarely exceeding 1.5%, even with up to 240 letters. Crucially, it autonomously performed end-to-end survival analysis, generating text-to-R-code that produced Kaplan-Meier curves statistically indistinguishable from ground truth. Consistent performance was demonstrated on a small validation cohort of 80 synthetic acute myeloid leukemia reports. Stress testing on data with simulated imperfections revealed a key role of a human-in-the-loop to resolve AI-flagged ambiguities. Furthermore, the LLM generated several correctly justified, biologically plausible, and potentially novel hypotheses from datasets up to 80 letters.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This feasibility study demonstrated that a frontier LLM (Gemini 2.5 Pro) can successfully perform high-fidelity data extraction, multiconditional querying, and automated survival analysis directly from unstructured text. These results provide a foundational proof of concept for the JIT clinical analysis approach. However, these findings are confined to synthetic patients, and rigorous validation on real-world clinical data is an essential next step before clinical implementation can be considered.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/78332","type":"journal-article","created":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T13:40:07Z","timestamp":1761226807000},"page":"e78332-e78332","source":"Crossref","is-referenced-by-count":5,"title":["Enabling Just-in-Time Clinical Oncology Analysis With Large Language Models: Feasibility and Validation Study Using Unstructured Synthetic Data"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9045-0595","authenticated-orcid":false,"given":"Peter","family":"May","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6153-3256","authenticated-orcid":false,"given":"Julian","family":"Gre\u00df","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6023-6336","authenticated-orcid":false,"given":"Christoph","family":"Seidel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7446-7629","authenticated-orcid":false,"given":"Sebastian","family":"Sommer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0764-1049","authenticated-orcid":false,"given":"Markus K","family":"Schuler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-9433-9123","authenticated-orcid":false,"given":"Sina","family":"Nokodian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0295-6932","authenticated-orcid":false,"given":"Florian","family":"Schr\u00f6der","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0137-7929","authenticated-orcid":false,"given":"Johannes","family":"Jung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,12,1]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-53190-9","article-title":"Generalizable and automated classification of TNM stage from pathology reports with external validation","volume":"15","author":"Kefeli","journal-title":"Nat Commun"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.23.00091","article-title":"Extraction of unstructured electronic health records to evaluate glioblastoma treatment patterns","volume":"8","author":"Swaminathan","journal-title":"JCO Clin Cancer Inform"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/64697","article-title":"A deep learning-enabled workflow to estimate real-world progression-free survival in patients with metastatic breast cancer: study using deidentified electronic health records","volume":"11","author":"Varma","journal-title":"JMIR Cancer"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-023-05439-1","article-title":"Empirical evaluation of language modeling to ascertain cancer outcomes from clinical text reports","volume":"24","author":"Elmarakeby","journal-title":"BMC Bioinformatics"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1089\/aipo.2024.0043","article-title":"Large language model extraction of PD-L1 biomarker testing details from electronic health records","volume":"2","author":"Cohen","journal-title":"AI Precision Oncol"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/65984","article-title":"Large language model applications for health information extraction in oncology: scoping review","volume":"11","author":"Chen","journal-title":"JMIR Cancer"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01233-2","article-title":"Privacy-preserving large language models for structured medical information retrieval","volume":"7","author":"Wiest","journal-title":"NPJ Digit Med"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/57275","article-title":"Large language model approach for zero-shot information extraction and clustering of Japanese radiology reports: algorithm development and validation","volume":"11","author":"Yamagishi","journal-title":"JMIR Cancer"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.2196\/72005","article-title":"Cancer diagnosis categorization in electronic health records using large language models and BioBERT: model performance evaluation study","volume":"11","author":"Hashtarkhani","journal-title":"JMIR Cancer"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/67801","article-title":"Systemic anticancer therapy timelines extraction from electronic medical records text: algorithm development and validation","volume":"6","author":"Yao","journal-title":"JMIR Bioinform Biotech"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/75556","article-title":"Automated esophageal cancer staging from free-text radiology reports: large language model evaluation study","volume":"13","author":"Yao","journal-title":"JMIR Med Inform"},{"issue":"6","key":"R12","doi-asserted-by":"publisher","DOI":"10.1056\/AIcs2300235","article-title":"GPT-4 for information retrieval and comparison of medical oncology guidelines","volume":"1","author":"Ferber","journal-title":"NEJM AI"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-81052-3","article-title":"Evaluation of the integration of retrieval-augmented generation in large language model for breast cancer nursing care responses","volume":"14","author":"Xu","journal-title":"Sci Rep"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Stuhlmiller TJ Rabe A Rapp J et al. A scalable method for validated data extraction from electronic health records with large language models. Health Informatics. 2025. [doi: 10.1101\/2025.02.25.25322898]","DOI":"10.1101\/2025.02.25.25322898"},{"issue":"7","key":"R15","doi-asserted-by":"publisher","first-page":"1952","DOI":"10.1109\/JBHI.2020.2990797","article-title":"Knowledge graph-enabled cancer data analytics","volume":"24","author":"Hasan","journal-title":"IEEE J Biomed Health Inform"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.2196\/64486","article-title":"Accuracy of large language models when answering clinical research questions: systematic review and network meta-analysis","volume":"27","author":"Wang","journal-title":"J Med Internet Res"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1533508","article-title":"Large language models generating synthetic clinical datasets: a feasibility and comparative analysis with real-world perioperative data","volume":"8","author":"Barr","journal-title":"Front Artif Intell"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.20.00184","article-title":"Clinical inflection point detection on the basis of EHR data to identify clinical trial-ready patients with cancer","volume":"5","author":"Kehl","journal-title":"JCO Clin Cancer Inform"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/68707","article-title":"Performance of natural language processing for information extraction from electronic health records within cancer: systematic review","volume":"13","author":"Dahl","journal-title":"JMIR Med Inform"},{"issue":"4","key":"R20","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","journal-title":"Nat Med"},{"issue":"6","key":"R21","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtocrr.2022.100340","article-title":"Automating access to real-world evidence","volume":"3","author":"Gauthier","journal-title":"JTO Clin Res Rep"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Elbattah M Arnaud E Ghazali DA Dequen G . Exploring the ethical challenges of large language models in emergency medicine: a comparative international review. In: Dequen G , editor. Presented at: 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); Dec 3-6, 2024; Lisbon, Portugal. 2024.[doi: 10.1109\/BIBM62325.2024.10822376]","DOI":"10.1109\/BIBM62325.2024.10822376"},{"issue":"6","key":"R23","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1192\/bjp.2024.134","article-title":"Detection of suicidality from medical text using privacy-preserving large language models","volume":"225","author":"Wiest","journal-title":"Br J Psychiatry"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/64544","article-title":"Using large language models to automate data extraction from surgical pathology reports: retrospective cohort study","volume":"9","author":"Lee","journal-title":"JMIR Form Res"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2400537","article-title":"Deidentifying medical documents with local, privacy-preserving large language models: the LLM-anonymizer","volume":"2","author":"Wiest","journal-title":"NEJM AI"},{"issue":"21","key":"R26","doi-asserted-by":"publisher","first-page":"5463","DOI":"10.1158\/0008-5472.CAN-19-0579","article-title":"Use of natural language processing to extract clinical cancer phenotypes from electronic medical records","volume":"79","author":"Savova","journal-title":"Cancer Res"},{"issue":"9","key":"R27","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1016\/j.ijmedinf.2014.06.009","article-title":"Text mining of cancer-related information: review of current status and future directions","volume":"83","author":"Spasi\u0107","journal-title":"Int J Med Inform"},{"key":"R28","unstructured":"Li M Huang J Yeung J Blaes A Johnson S Liu H et al. CancerLLM: a large language model in cancer domain. arXiv. Preprint posted online on Jun 1, 2024. URL: https:\/\/ui.adsabs.harvard.edu\/abs\/2024arXiv240610459L [Accessed 14-11-2025]"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/59069","article-title":"Revolutionizing health care: the transformative impact of large language models in medicine","volume":"27","author":"Zhang","journal-title":"J Med Internet Res"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/j.lanepe.2024.101064","article-title":"Artificial intelligence in oncology: ensuring safe and effective integration of language models in clinical practice","volume":"46","author":"Verlingue","journal-title":"Lancet Reg Health Eur"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01559-5","article-title":"Examining human-AI interaction in real-world healthcare beyond the laboratory","volume":"8","author":"Wekenborg","journal-title":"NPJ Digit Med"},{"key":"R32","unstructured":"JIT-AI-registry. GitHub. URL: https:\/\/github.com\/pcmay\/JIT-AI-registry [Accessed 04-11-2025]"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e78332","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:10:55Z","timestamp":1782843055000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e78332"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,1]]},"references-count":32,"URL":"https:\/\/doi.org\/10.2196\/78332","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,1]]},"article-number":"v13i1e78332"}}