{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T19:07:09Z","timestamp":1788894429305,"version":"build-2803163510"},"reference-count":35,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Perioperative computed tomography (CT) imaging is essential for detecting postoperative recurrence and metastasis in cancer patients. However, large-scale automated extraction of oncological outcomes from CT reports remains limited by the unstructured nature of report text and wide variability in reporting styles. Radiology reports frequently contain linguistic ambiguities, including negations, hedging, and expressions conveying diagnostic uncertainty (eg, \u201ccannot exclude recurrence\u201d or \u201cpossibly metastatic\u201d). Manual review is labor-intensive and constrains consistent extraction at scale. The inability to systematically account for diagnostic uncertainty represents a major barrier to reliable automated surveillance systems.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop a semisupervised deep learning (DL) classification framework that explicitly captures diagnostic uncertainty by classifying postoperative recurrence and metastasis into 3 categories (positive, negative, and uncertain).<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This retrospective study analyzed 288,076 postoperative CT reports from 86,083 cancer surgery patients at Asan Medical Center (2014\u20102021). After exact-match deduplication, model training and evaluation used 17,846 unique reports for recurrence and 63,766 for metastasis. Preprocessing identified presumed negatives through keyword filtering and unsupervised clustering. The semisupervised framework incorporated human-in-the-loop validation across 3 cycles\u2014with clinicians reviewing approximately 2000 samples per cycle (<1% of total reports)\u2014and integrated rule-based algorithms (RAs) and medical BERT (MedEmbed and PubMedBERT). A report-level train-validation split was used, as preprocessing reduces each report to sentence-level fragments that preclude patient-level linkage. Performance was evaluated against RAs and multiple BERT variants under both naive and simulated real-world class distributions. Maximum mean discrepancy testing confirmed distributional integrity of the sampled data. Model interpretability was assessed using Integrated Gradients.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The cohort included 11 cancer types, predominantly gastrointestinal (28,516\/86,083, 33.1%), hepatobiliary and pancreas (14,715\/86,083, 17.1%), and genitourinary (12,959\/86,083, 15.1%). Under simulated conditions, PubMedBERT achieved 92.58% accuracy for recurrence in the multiclass, and MedEmbed achieved 93.25% for metastasis in the binary class. The framework achieved accuracies of 97.33% (multiclass) and 99.33% (binary class) for recurrence and 95.00% (multiclass) and 96.67% (binary class) for metastasis, compared with human intrarater consistencies of 96.88% and 93.80% (recurrence and metastasis, respectively, for multiclass), reflecting concordance with the clinician-derived consensus standard. The framework captured diagnostic uncertainty in 1.4% of recurrence cases and 6.9% of metastasis cases. Notably, the RA outperformed several sophisticated DL models in metastasis classification.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The proposed framework achieves clinician-concordant classification across the full 288,076-report corpus while requiring minimal expert annotation (<1% of reports). By explicitly modeling diagnostic uncertainty and combining rule-based and DL approaches, it demonstrates the potential for automated cancer surveillance and clinical decision support in real-world settings; however, generalizability to other institutions requires prospective multicenter validation.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/92937","type":"journal-article","created":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T18:23:58Z","timestamp":1788891838000},"page":"e92937-e92937","source":"Crossref","is-referenced-by-count":0,"title":["Automated Extraction of Postoperative Cancer Recurrence and Metastasis From Computed Tomography (CT) Reports: Semisupervised Deep Learning Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7374-2900","authenticated-orcid":false,"given":"Wonkeun","family":"Jo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1651-364X","authenticated-orcid":false,"given":"Bumwoo","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7790-1961","authenticated-orcid":false,"given":"Ji-Hoon","family":"Sim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,8]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1148\/radiol.16142770","article-title":"Natural language processing in radiology: a systematic review","volume":"279","author":"Pons","journal-title":"Radiology"},{"issue":"11","key":"R2","doi-asserted-by":"publisher","first-page":"1185","DOI":"10.1016\/j.crad.2015.06.080","article-title":"Radiology reports: a quantifiable and objective textual approach","volume":"70","author":"Scott","journal-title":"Clin Radiol"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232085","article-title":"Weakly supervised deep learning in radiology","volume":"312","author":"Misera","journal-title":"Radiology"},{"issue":"7","key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/27955","article-title":"Automatic extraction of lung cancer staging information from computed tomography reports: deep learning approach","volume":"9","author":"Hu","journal-title":"JMIR Med Inform"},{"issue":"12","key":"R5","doi-asserted-by":"publisher","first-page":"100926","DOI":"10.1016\/j.landig.2025.100926","article-title":"Development and validation of a pre-trained language model for neonatal morbidities: a retrospective, multicentre, prognostic study","volume":"7","author":"Xie","journal-title":"Lancet Digit Health"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/18143","article-title":"Incorporating breast cancer recurrence events into population-based cancer registries using medical claims: cohort study","volume":"6","author":"A\u2019mar","journal-title":"JMIR Cancer"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.913","article-title":"Negation and uncertainty detection in clinical texts written in Spanish: a deep learning-based approach","volume":"8","author":"Solarte Pab\u00f3n","journal-title":"PeerJ Comput Sci"},{"issue":"5","key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/24381","article-title":"Automating stroke data extraction from free-text radiology reports using natural language processing: instrument validation study","volume":"9","author":"Yu","journal-title":"JMIR Med Inform"},{"issue":"5","key":"R9","doi-asserted-by":"publisher","first-page":"1194","DOI":"10.1007\/s10278-020-00379-1","article-title":"Between always and never: evaluating uncertainty in radiology reports using natural language processing","volume":"33","author":"Callen","journal-title":"J Digit Imaging"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1160167","article-title":"A scoping review of natural language processing of radiology reports in breast cancer","volume":"13","author":"Saha","journal-title":"Front Oncol"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","first-page":"1041","DOI":"10.1158\/2767-9764.CRC-24-0064","article-title":"Artificial intelligence-assisted cancer status detection in radiology reports","volume":"4","author":"Arya","journal-title":"Cancer Res Commun"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaf070","article-title":"Large-scale deep learning for metastasis detection in pathology reports","volume":"8","author":"Krawczuk","journal-title":"JAMIA Open"},{"issue":"4","key":"R13","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","journal-title":"Bioinformatics"},{"issue":"11","key":"R14","doi-asserted-by":"publisher","first-page":"1297","DOI":"10.1093\/jamia\/ocz096","article-title":"Enhancing clinical concept extraction with contextual embeddings","volume":"26","author":"Si","journal-title":"J Am Med Inform Assoc"},{"key":"R15","unstructured":"van den Oord A Li Y Vinyals O . Representation learning with contrastive predictive coding. arXiv. Preprint posted online on 2018. [doi: 10.48550\/arXiv.1807.03748]"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104576","article-title":"Deep learning uncertainty quantification for clinical text classification","volume":"149","author":"Peluso","journal-title":"J Biomed Inform"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s43621-024-00218-1","article-title":"A semi-supervised approach of short text topic modeling using embedded fuzzy clustering for Twitter hashtag recommendation","volume":"5","author":"Pattanayak","journal-title":"Discov Sustain"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","DOI":"10.3390\/cancers14040898","article-title":"UMAP based anomaly detection for minimal residual disease quantification within acute myeloid leukemia","volume":"14","author":"Weijler","journal-title":"Cancers (Basel)"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","first-page":"144","DOI":"10.3390\/computation13060144","article-title":"Scalable clustering of complex ECG health data: big data clustering analysis with UMAP and HDBSCAN","volume":"13","author":"Kaverinskiy","journal-title":"Computation"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2024.100664","article-title":"A comprehensive survey of text classification techniques and their research applications: observational and experimental insights","volume":"54","author":"Taha","journal-title":"Computer Science Review"},{"key":"R21","unstructured":"Ester M Kriegel HP Sander J Xu X . A density-based algorithm for discovering clusters in large spatial databases with noise. Presented at: Second International Conference on Knowledge Discovery and Data Mining. 1996.[doi: 10.5555\/3001460.3001507]"},{"key":"R22","unstructured":"MedEmbed is a collection of embedding models fine-tuned specifically for medical and clinical data. GitHub. 2024. URL: https:\/\/github.com\/abhinand5\/MedEmbed [Accessed 01-08-2026]"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing","volume":"3","author":"Gu","journal-title":"ACM Trans Comput Healthcare"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","article-title":"Human-in-the-loop machine learning: a state of the art","volume":"56","author":"Mosqueira-Rey","journal-title":"Artif Intell Rev"},{"issue":"25","key":"R25","first-page":"723","volume":"13","author":"Gretton","journal-title":"J Mach Learn Res"},{"key":"R26","unstructured":"Sundararajan M Taly A Yan Q . Axiomatic attribution for deep networks. Presented at: 34th International Conference on Machine Learning. 2017.[doi: 10.5555\/3305890.3306024]"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1007\/s10278-017-0027-x","article-title":"Integrating Natural Language Processing and Machine Learning Algorithms to Categorize Oncologic Response in Radiology Reports","volume":"31","author":"Chen","journal-title":"J Digit Imaging"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02267-w","article-title":"Clinically informed semi-supervised learning improves disease annotation and equity from electronic health records: a glaucoma case study","volume":"9","author":"Moradi","journal-title":"NPJ Digit Med"},{"issue":"10","key":"R29","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-021-01761-4","article-title":"Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance","volume":"45","author":"Olthof","journal-title":"J Med Syst"},{"issue":"5","key":"R30","doi-asserted-by":"publisher","first-page":"e428","DOI":"10.1016\/S2468-2667(25)00036-2","article-title":"Artificial intelligence in public health: promises, challenges, and an agenda for policy makers and public health institutions","volume":"10","author":"Panteli","journal-title":"Lancet Public Health"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1016\/j.eclinm.2025.103252","article-title":"Clinicians must participate in the development of multimodal AI","volume":"84","author":"Banerji","journal-title":"EClinicalMedicine"},{"issue":"3","key":"R32","doi-asserted-by":"publisher","first-page":"837","DOI":"10.1038\/s41591-024-02850-w","article-title":"Heterogeneity and predictors of the effects of AI assistance on radiologists","volume":"30","author":"Yu","journal-title":"Nat Med"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01522-4","article-title":"Weakly supervised language models for automated extraction of critical findings from radiology reports","volume":"8","author":"Das","journal-title":"NPJ Digit Med"},{"key":"R34","doi-asserted-by":"crossref","unstructured":"Baumann N Kusmenko E Ritz J Rumpe B Weber MB . Dynamic data management for continuous retraining. Presented at: MODELS \u201922; Montreal Quebec Canada. Oct 23, 2022.URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3550356 [doi: 10.1145\/3550356.3561568]","DOI":"10.1145\/3550356.3561568"},{"key":"R35","unstructured":"merchen911\/postoperative_cancer_labeling. GitHub. URL: https:\/\/github.com\/merchen911\/postoperative_cancer_labeling [Accessed 01-08-2026]"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T18:24:00Z","timestamp":1788891840000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e92937"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,8]]},"references-count":35,"URL":"https:\/\/doi.org\/10.2196\/92937","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,8]]},"article-number":"v14i2e92937"}}