{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:09Z","timestamp":1785544989372,"version":"3.56.0"},"reference-count":35,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Lymph node metastasis (LNM) is a critical clinical indicator for determining the initial treatment strategy for patients with lung cancer. However, accurately diagnosing LNM preoperatively remains a significant challenge. Data-driven predictive modeling has become a mainstream approach to address this issue, yet it often overlooks existing clinical knowledge. Large language models (LLMs) have demonstrated the potential to predict clinical risks in a zero-shot manner based on the extensive clinical knowledge learned from large-scale corpora.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n LLMs have demonstrated the potential to predict clinical risks in a zero-shot manner based on the extensive clinical knowledge learned from large-scale corpora. This study aims to investigate the integration of LLM-derived knowledge with data-driven patterns to enhance the accuracy of LNM prediction.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We propose a novel ensemble framework that combines the strengths of LLMs and machine learning (ML) models for LNM prediction in lung cancer. Specifically, 3 ML models were trained using clinical data, and their predicted probabilities, along with the original clinical features, were incorporated into prompts for LLMs. Three LLMs\u2014GPT-5.4, GPT-5.4-nano, and DeepSeek-V3.2\u2014were used to independently predict LNM risk 5 times, and 4 ensemble strategies were applied to aggregate their predictions into a final outcome.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The proposed approach was evaluated on clinical data from 767 patients with lung cancer at Peking University Cancer Hospital. Experimental results show that our proposed framework significantly outperforms base ML models, achieving an area under the curve of 0.781 and an average precision of 0.420. Compared with the no reasoning English setting, both the reasoning English setting and nonreasoning Chinese setting showed a lower area under the curve but higher average precision.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study presents a novel knowledge-augmented strategy for integrating the clinical knowledge embedded in LLMs with the statistical patterns captured by ML models to improve the LNM prediction of lung cancer, offering a new paradigm for integrating medical knowledge and patient data in clinical predictions.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/86700","type":"journal-article","created":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T21:16:55Z","timestamp":1782163015000},"page":"e86700-e86700","source":"Crossref","is-referenced-by-count":0,"title":["Leveraging Large Language Models to Integrate Clinical Knowledge and Machine Learning Predictions for Lymph Node Metastasis Prediction: Development of a Knowledge-Augmented Framework"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-3150-3561","authenticated-orcid":false,"given":"Hongying","family":"Yu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8271-4530","authenticated-orcid":false,"given":"Bing","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7405-6992","authenticated-orcid":false,"given":"Xian","family":"Zeng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6825-0757","authenticated-orcid":false,"given":"Mucheng","family":"Ren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4934-0888","authenticated-orcid":false,"given":"Zheng","family":"Cao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9688-104X","authenticated-orcid":false,"given":"Xiaofeng","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7658-5250","authenticated-orcid":false,"given":"Xudong","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5315-8811","authenticated-orcid":false,"given":"Jun","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3531-1656","authenticated-orcid":false,"given":"Nan","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0810-4819","authenticated-orcid":false,"given":"Danqing","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,22]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"209","DOI":"10.3322\/caac.21660","article-title":"Global Cancer Statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"71","author":"Sung","journal-title":"CA Cancer J Clin"},{"issue":"5 Suppl","key":"R2","doi-asserted-by":"publisher","first-page":"e278S","DOI":"10.1378\/chest.12-2359","article-title":"Treatment of stage I and II non-small cell lung cancer: diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines","volume":"143","author":"Howington","journal-title":"Chest"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1016\/j.chest.2018.10.020","article-title":"The accuracy of clinical staging of stage I-IIIa non-small cell lung cancer: an analysis based on individual participant data","volume":"155","author":"Navani","journal-title":"Chest"},{"issue":"9","key":"R4","doi-asserted-by":"publisher","first-page":"1170","DOI":"10.1097\/JTO.0b013e3182992421","article-title":"A prediction model for pathologic N2 disease in lung cancer patients with a negative mediastinum by positron emission tomography","volume":"8","author":"Farjah","journal-title":"J Thorac Oncol"},{"issue":"5","key":"R5","doi-asserted-by":"publisher","first-page":"1761","DOI":"10.1016\/j.athoracsur.2013.06.038","article-title":"Development and validation of a clinical prediction model for N2 lymph node metastasis in non-small cell lung cancer","volume":"96","author":"Chen","journal-title":"Ann Thorac Surg"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.athoracsur.2018.02.026","article-title":"A texture analysis-based prediction model for lymph node metastasis in stage Ia lung adenocarcinoma","volume":"106","author":"Gu","journal-title":"Ann Thorac Surg"},{"issue":"4","key":"R7","doi-asserted-by":"publisher","first-page":"641","DOI":"10.21147\/j.issn.1000-9604.2019.04.08","article-title":"Radiomics-based predictive risk score: a scoring system for preoperatively predicting risk of lymph node metastasis in patients with resectable non-small cell lung cancer","volume":"31","author":"He","journal-title":"Chin J Cancer Res"},{"issue":"11","key":"R8","doi-asserted-by":"publisher","first-page":"6049","DOI":"10.1007\/s00330-019-06084-0","article-title":"Can peritumoral radiomics increase the efficiency of the prediction for lymph node metastasis in clinical stage T1 lung adenocarcinoma on CT?","volume":"29","author":"Wang","journal-title":"Eur Radiol"},{"issue":"10","key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtho.2018.08.357","article-title":"MA05.11 radiomics analysis using SVM predicts mediastinal lymph nodes status of Squamous Cell Lung Cancer by pre-treatment chest CT scan","volume":"13","author":"Wang","journal-title":"J Thorac Oncol"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.lungcan.2019.11.003","article-title":"Development of a predictive radiomics model for lymph node metastases in pre-surgical CT-based stage IA non-small cell lung cancer","volume":"139","author":"Cong","journal-title":"Lung Cancer"},{"issue":"6","key":"R11","doi-asserted-by":"publisher","first-page":"4184","DOI":"10.1007\/s00330-020-07523-z","article-title":"Machine learning-based diagnostic method of pre-therapeutic F-FDG PET\/CT for evaluating mediastinal lymph nodes in non-small cell lung cancer","volume":"31","author":"Yoo","journal-title":"Eur Radiol"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/35475","article-title":"Using natural language processing and machine learning to preoperatively predict lymph node metastasis for non-small cell lung cancer with electronic medical records: development and validation study","volume":"10","author":"Hu","journal-title":"JMIR Med Inform"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.lungcan.2020.04.014","article-title":"A cross-modal 3D deep learning for accurate lymph node metastasis prediction in clinical stage T1 lung adenocarcinoma","volume":"145","author":"Zhao","journal-title":"Lung Cancer"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1186\/s13550-017-0260-9","article-title":"Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from F-FDG PET\/CT images","volume":"7","author":"Wang","journal-title":"EJNMMI Res"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.compmedimag.2021.101935","article-title":"Dual energy CT image prediction on primary tumor of lung cancer for nodal metastasis using deep learning","volume":"91","author":"Wang","journal-title":"Comput Med Imaging Graph"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","first-page":"1216","DOI":"10.1109\/JBHI.2022.3233387","article-title":"A multi-modal heterogeneous graph forest to predict lymph node metastasis of non-small cell lung cancer","volume":"27","author":"Hu","journal-title":"IEEE J Biomed Health Inform"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI231096","article-title":"A deep multi-task network to learn tumor pathological representations for lymph node metastasis prediction","volume":"310","author":"Hu","journal-title":"Stud Health Technol Inform"},{"key":"R18","unstructured":"Introducing ChatGPT. OpenAI. URL: https:\/\/openai.com\/blog\/chatgpt [Accessed 09-06-2026]"},{"key":"R19","unstructured":"Achiam J Adler S Agarwal S et al. Gpt-4 technical report. arXiv. Preprint posted online on Mar 15, 2023. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R20","unstructured":"Brown TB Mann B Ryder N et al. Language models are few-shot learners. Presented at: NIPS\u201920: the 34th International Conference on Neural Information Processing Systems; Dec 6-12, 2020; Vancouver, BC, Canada. URL: https:\/\/dl.acm.org\/doi\/abs\/10.5555\/3495724.3495883 [Accessed 09-06-2026]"},{"key":"R21","unstructured":"Ouyang L Wu J Jiang X et al. Training language models to follow instructions with human feedback. Presented at: 36th Conference on Neural Information Processing Systems (NeurIPS 2022); Nov 28 to Dec 9, 2022; New Orleans, Louisiana, USA. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/b1efde53be364a73914f58805a001731-Paper-Conference.pdf [Accessed 09-06-2026]"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00896-7","article-title":"Evaluating large language models on medical evidence summarization","volume":"6","author":"Tang","journal-title":"NPJ Digit Med"},{"issue":"9","key":"R23","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R24","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231593","article-title":"Quantitative evaluation of large language models to streamline radiology report impressions: a multimodal retrospective analysis","volume":"310","author":"Doshi","journal-title":"Radiology"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Hu D Zhang S Liu Q Zhu X Liu B . The current status of large language models in summarizing radiology report impressions. arXiv. Preprint posted online on Jun 4, 2024. [doi: 10.2196\/preprints.65547]","DOI":"10.2196\/preprints.65547"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2023.105321","article-title":"Zero-shot information extraction from radiological reports using ChatGPT","volume":"183","author":"Hu","journal-title":"Int J Med Inform"},{"issue":"8","key":"R27","doi-asserted-by":"publisher","first-page":"928","DOI":"10.1001\/jamasurg.2024.1621","article-title":"Large language model capabilities in perioperative risk prediction and prognostication","volume":"159","author":"Chung","journal-title":"JAMA Surg"},{"issue":"9","key":"R28","doi-asserted-by":"publisher","first-page":"1921","DOI":"10.1093\/jamia\/ocae103","article-title":"Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room","volume":"31","author":"Glicksberg","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2024.109022","article-title":"Evaluation of GPT-4 for 10-year cardiovascular risk prediction: insights from the UK Biobank and KoGES data","volume":"27","author":"Han","journal-title":"iScience"},{"key":"R30","unstructured":"Zhu Y Wang Z Gao J et al. Prompting large language models for zero-shot clinical prediction with structured longitudinal electronic health record data. arXiv. Preprint posted online on Jan 25, 2024. [doi: 10.48550\/arXiv.2402.01713]"},{"issue":"7","key":"R31","doi-asserted-by":"publisher","DOI":"10.2196\/27955","article-title":"Automatic extraction of lung cancer staging information from computed tomography reports: deep learning approach","volume":"9","author":"Hu","journal-title":"JMIR Med Inform"},{"key":"R32","unstructured":"Yan Z Zhang K Zhou R He L Li X Sun L . Multimodal ChatGPT for medical applications: an experimental study of GPT-4V. arXiv. Preprint posted online on Oct 29, 2023. [doi: 10.48550\/arXiv.2310.19061]"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/54393","article-title":"Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: evaluation study","volume":"10","author":"Nakao","journal-title":"JMIR Med Educ"},{"issue":"2","key":"R34","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.233270","article-title":"Evaluating GPT-V4 (GPT-4 with Vision) on detection of radiologic findings on chest radiographs","volume":"311","author":"Zhou","journal-title":"Radiology"},{"issue":"4","key":"R35","doi-asserted-by":"publisher","first-page":"1959","DOI":"10.1007\/s00330-024-11035-5","article-title":"Assessing GPT-4 multimodal performance in radiological image analysis","volume":"35","author":"Brin","journal-title":"Eur Radiol"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e86700","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:25Z","timestamp":1785536725000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e86700"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,22]]},"references-count":35,"URL":"https:\/\/doi.org\/10.2196\/86700","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,22]]},"article-number":"v14i3e86700"}}