{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T05:19:20Z","timestamp":1787807960925,"version":"build-2784847793"},"reference-count":29,"publisher":"American Society of Clinical Oncology (ASCO)","issue":"8","content-domain":{"domain":["ascopubs.org"],"crossmark-restriction":true},"short-container-title":["JCO Clin Cancer Inform"],"published-print":{"date-parts":[[2024,5]]},"abstract":"<jats:sec><jats:title>PURPOSE<\/jats:title><jats:p> To evaluate natural language processing (NLP) methods to infer metastatic sites from radiology reports. <\/jats:p><\/jats:sec><jats:sec><jats:title>METHODS<\/jats:title><jats:p> A set of 4,522 computed tomography (CT) reports of 550 patients with 14 types of cancer was used to fine-tune four clinical large language models (LLMs) for multilabel classification of metastatic sites. We also developed an NLP information extraction (IE) system (on the basis of named entity recognition, assertion status detection, and relation extraction) for comparison. Model performances were measured by F1 scores on test and three external validation sets. The best model was used to facilitate analysis of metastatic frequencies in a cohort study of 6,555 patients with 53,838 CT reports. <\/jats:p><\/jats:sec><jats:sec><jats:title>RESULTS<\/jats:title><jats:p> The RadBERT, BioBERT, GatorTron-base, and GatorTron-medium LLMs achieved F1 scores of 0.84, 0.87, 0.89, and 0.91, respectively, on the test set. The IE system performed best, achieving an F1 score of 0.93. F1 scores of the IE system by individual cancer type ranged from 0.89 to 0.96. The IE system attained F1 scores of 0.89, 0.83, and 0.81, respectively, on external validation sets including additional cancer types, positron emission tomography-CT ,and magnetic resonance imaging scans, respectively. In our cohort study, we found that for colorectal cancer, liver-only metastases were higher in de novo stage IV versus recurrent patients (29.7% v 12.2%; P &lt; .001). Conversely, lung-only metastases were more frequent in recurrent versus de novo stage IV patients (17.2% v 7.3%; P &lt; .001). <\/jats:p><\/jats:sec><jats:sec><jats:title>CONCLUSION<\/jats:title><jats:p> We developed an IE system that accurately infers metastatic sites in multiple primary cancers from radiology reports. It has explainable methods and performs better than some clinical LLMs. The inferred metastatic phenotypes could enhance cancer research databases and clinical trial matching, and identify potential patients for oligometastatic interventions. <\/jats:p><\/jats:sec>","DOI":"10.1200\/cci.23.00122","type":"journal-article","created":{"date-parts":[[2024,5,24]],"date-time":"2024-05-24T19:59:36Z","timestamp":1716580776000},"update-policy":"https:\/\/doi.org\/10.1200\/crossmark","source":"Crossref","is-referenced-by-count":10,"title":["Use of Natural Language Processing to Infer Sites of Metastatic Disease From Radiology Reports at Scale"],"prefix":"10.1200","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1299-0668","authenticated-orcid":true,"given":"See Boon","family":"Tay","sequence":"first","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"NUS Yong Loo Lin School of Medicine, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1768-807X","authenticated-orcid":true,"given":"Guat Hwa","family":"Low","sequence":"additional","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gillian Jing En","family":"Wong","sequence":"additional","affiliation":[{"name":"NUS Yong Loo Lin School of Medicine, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Han Jieh","family":"Tey","sequence":"additional","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2352-9076","authenticated-orcid":true,"given":"Fun Loon","family":"Leong","sequence":"additional","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5487-7225","authenticated-orcid":true,"given":"Constance","family":"Li","sequence":"additional","affiliation":[{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1648-1473","authenticated-orcid":true,"given":"Melvin Lee Kiang","family":"Chua","sequence":"additional","affiliation":[{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Singapore Duke-NUS Medical School, Singapore, Singapore"},{"name":"Division of Radiation Oncology, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6514-6786","authenticated-orcid":true,"given":"Daniel Shao Weng","family":"Tan","sequence":"additional","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Singapore Duke-NUS Medical School, Singapore, Singapore"},{"name":"Division of Clinical Trials and Epidemiological Sciences, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Choon Hua","family":"Thng","sequence":"additional","affiliation":[{"name":"Singapore Duke-NUS Medical School, Singapore, Singapore"},{"name":"Division of Oncologic Imaging, National Cancer Centre Singapore, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iain Bee Huat","family":"Tan","sequence":"additional","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Singapore Duke-NUS Medical School, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1023-5730","authenticated-orcid":true,"given":"Ryan Shea Ying Cong","family":"Tan","sequence":"additional","affiliation":[{"name":"Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Data and Computational Science Core, National Cancer Centre Singapore, Singapore, Singapore"},{"name":"Singapore Duke-NUS Medical School, Singapore, Singapore"},{"name":"Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"233","reference":[{"key":"b1","doi-asserted-by":"publisher","DOI":"10.1186\/s12885-018-4371-0"},{"key":"b2","doi-asserted-by":"publisher","DOI":"10.1002\/cam4.2474"},{"key":"b3","doi-asserted-by":"publisher","DOI":"10.1038\/s41392-020-0134-x"},{"key":"b4","doi-asserted-by":"publisher","DOI":"10.1016\/j.devcel.2019.04.012"},{"key":"b5","doi-asserted-by":"publisher","DOI":"10.3390\/cells10112944"},{"key":"b6","doi-asserted-by":"publisher","DOI":"10.18632\/oncotarget.2677"},{"key":"b7","first-page":"90","volume":"28","author":"Krug D","year":"2021","journal-title":"Clin Transl Radiat Oncol"},{"key":"b8","doi-asserted-by":"publisher","DOI":"10.1016\/j.lungcan.2010.05.003"},{"key":"b9","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2021210043"},{"key":"b10","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.16142770"},{"key":"b11","author":"Karimi YH","journal-title":"JCO Clin Cancer Inform"},{"key":"b12","author":"Sun S","journal-title":"JCO Clin Cancer Inform"},{"key":"b13","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.210258"},{"key":"b14","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz682"},{"key":"b15","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00742-2"},{"key":"b17","doi-asserted-by":"publisher","DOI":"10.1186\/s12894-015-0087-5"},{"key":"b18","doi-asserted-by":"publisher","DOI":"10.1164\/rccm.201501-0017OC"},{"key":"b21","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2021.100058"},{"key":"b22","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-018-0116-5"},{"key":"b23","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00104"},{"key":"b28","doi-asserted-by":"publisher","DOI":"10.1001\/jamaoncol.2019.1800"},{"key":"b31","doi-asserted-by":"publisher","DOI":"10.5858\/2008-132-931-MPOCRF"},{"key":"b32","doi-asserted-by":"publisher","DOI":"10.4240\/wjgs.v13.i8.756"},{"key":"b33","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2014.12.003"},{"key":"b34","author":"Kirshner J","journal-title":"JCO Clin Cancer Inform"},{"key":"b35","author":"Jain NM","journal-title":"JCO Clin Cancer Inform"},{"key":"b36","doi-asserted-by":"publisher","DOI":"10.1158\/0008-5472.CAN-19-0579"},{"key":"b37","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaa002"},{"key":"b38","doi-asserted-by":"publisher","DOI":"10.2196\/27767"}],"container-title":["JCO Clinical Cancer Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ascopubs.org\/doi\/pdfdirect\/10.1200\/CCI.23.00122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,24]],"date-time":"2024-05-24T19:59:44Z","timestamp":1716580784000},"score":1,"resource":{"primary":{"URL":"https:\/\/ascopubs.org\/doi\/10.1200\/CCI.23.00122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5]]},"references-count":29,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,5]]}},"alternative-id":["10.1200\/CCI.23.00122"],"URL":"https:\/\/doi.org\/10.1200\/cci.23.00122","relation":{},"ISSN":["2473-4276"],"issn-type":[{"value":"2473-4276","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5]]},"assertion":[{"value":"2023-07-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-02","order":1,"name":"revised","label":"Revised","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2300122"}}