{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T15:46:14Z","timestamp":1777391174557,"version":"3.51.4"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T00:00:00Z","timestamp":1756857600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["52361145714"],"award-info":[{"award-number":["52361145714"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["21673252"],"award-info":[{"award-number":["21673252"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002888","name":"Beijing Municipal Education Commission","doi-asserted-by":"publisher","award":["2019821001"],"award-info":[{"award-number":["2019821001"]}],"id":[{"id":"10.13039\/501100002888","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Immunotherapy has emerged as a promising treatment for advanced non-small cell lung cancer (NSCLC), but accurately predicting which patients will benefit from it remains a major clinical challenge. To address this, we aim to develop a novel multimodal method, DeepAFM, that integrates histopathology, genomic features, and clinical information to predict patient responses to anti-PD-(L)1 immunotherapy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>A total of 93 patients with advanced NSCLC were included in this study. Histopathological whole-slide images were processed using a self-supervised VQVAE2 for representation learning. PCA and K-means clustering were then applied for dimensionality reduction and feature grouping. Key regions of interest were visualized through permutation importance evaluation and color-coding techniques. The extracted histopathological features, along with genomic alterations and clinical variables, were integrated into the DeepAFM multimodal prediction model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The DeepAFM achieved a high predictive performance with an area under the curve (AUC) of 0.77 (95% confidence interval: 0.69-1.00). Attention-based heatmaps revealed that the model could identify critical pathological patterns, genomic mutations, and clinical indicators associated with patient responses to immunotherapy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The integration of multimodal data enabled the model to capture complex interactions among pathology, genomics, and clinical characteristics, enhancing the interpretability and predictive power of immunotherapy response prediction. The visualization techniques facilitated the identification of biologically meaningful features and potential biomarkers.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This study demonstrates the effectiveness of the DeepAFM in predicting responses to immunotherapy in advanced NSCLC. The approach not only improves prediction accuracy but also provides valuable insights for personalized treatment strategies and biomarker discovery.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf142","type":"journal-article","created":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T11:43:08Z","timestamp":1754134988000},"page":"1641-1653","source":"Crossref","is-referenced-by-count":1,"title":["Multimodal deep learning for immunotherapy response prediction and biomarker discovery in non-small cell lung cancer"],"prefix":"10.1093","volume":"32","author":[{"given":"Zijun","family":"Wang","sequence":"first","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]},{"given":"Xi","family":"Liu","sequence":"additional","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]},{"given":"Kaitai","family":"Han","sequence":"additional","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]},{"given":"Lixin","family":"Lei","sequence":"additional","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7595-5556","authenticated-orcid":false,"given":"Chaojing","family":"Shi","sequence":"additional","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]},{"given":"Wu","family":"Liu","sequence":"additional","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8895-7899","authenticated-orcid":false,"given":"Qianjin","family":"Guo","sequence":"additional","affiliation":[{"name":"Academy of Artificial Intelligence, Beijing Institute of Petrochemical Technology , 19 Qingyuan North Road, Daxing District , Beijing 102617,","place":["China"]}]}],"member":"286","published-online":{"date-parts":[[2025,9,3]]},"reference":[{"key":"2025111813571029800_ocaf142-B1","doi-asserted-by":"crossref","first-page":"1727","DOI":"10.1016\/S0140-6736(10)62101-0","article-title":"Non-small-cell lung cancer","volume":"378","author":"Goldstraw","year":"2011","journal-title":"Lancet"},{"key":"2025111813571029800_ocaf142-B2","doi-asserted-by":"publisher","first-page":"1432","DOI":"10.1126\/science.342.6165.1432","article-title":"Cancer immunotherapy","volume":"342","author":"Couzin-Frankel","year":"2013","journal-title":"Science"},{"key":"2025111813571029800_ocaf142-B3","doi-asserted-by":"publisher","first-page":"2018","DOI":"10.1056\/NEJMoa1501824","article-title":"Pembrolizumab for the treatment of non-small-cell lung cancer","volume":"372","author":"Garon","year":"2015","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B4","doi-asserted-by":"publisher","first-page":"1627","DOI":"10.1056\/NEJMoa1507643","article-title":"Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancer","volume":"373","author":"Borghaei","year":"2015","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B5","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1056\/NEJMoa1302369","article-title":"Nivolumab plus ipilimumab in advanced melanoma","volume":"369","author":"Wolchok","year":"2013","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B6","doi-asserted-by":"publisher","first-page":"2521","DOI":"10.1056\/NEJMoa1503093","article-title":"Pembrolizumab versus ipilimumab in advanced melanoma","volume":"372","author":"Robert","year":"2015","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B7","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1056\/NEJMoa2109970","article-title":"Relatlimab and nivolumab versus nivolumab in untreated advanced melanoma","volume":"386","author":"Tawbi","year":"2022","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B8","doi-asserted-by":"publisher","first-page":"2020","DOI":"10.1056\/NEJMoa1910231","article-title":"Nivolumab plus ipilimumab in advanced non-small-cell lung cancer","volume":"381","author":"Hellmann","year":"2019","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B9","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1056\/NEJMoa1411087","article-title":"PD-1 blockade with nivolumab in relapsed or refractory Hodgkin\u2019s lymphoma","volume":"372","author":"Ansell","year":"2015","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B10","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1634\/theoncologist.2017-0004","article-title":"FDA approval summary: nivolumab for the treatment of relapsed or progressive classical Hodgkin lymphoma","volume":"22","author":"Kasamon","year":"2017","journal-title":"Oncologist"},{"key":"2025111813571029800_ocaf142-B11","doi-asserted-by":"publisher","first-page":"1856","DOI":"10.1056\/NEJMoa1602252","article-title":"Nivolumab for recurrent squamous-cell carcinoma of the head and neck","volume":"375","author":"Ferris","year":"2016","journal-title":"N Engl J Med"},{"key":"2025111813571029800_ocaf142-B12","doi-asserted-by":"crossref","first-page":"672","DOI":"10.3389\/fcell.2020.00672","article-title":"Resistance mechanisms of anti-PD1\/PDL1 therapy in solid tumors","volume":"8","author":"Lei","year":"2020","journal-title":"Front Cell Dev Biol"},{"key":"2025111813571029800_ocaf142-B13","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1186\/s12943-020-1144-6","article-title":"Predictive biomarkers and mechanisms underlying resistance to PD1\/PD-L1 blockade cancer immunotherapy","volume":"19","author":"Ren","year":"2020","journal-title":"Mol Cancer"},{"key":"2025111813571029800_ocaf142-B14","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1586\/14737159.2016.1117389","article-title":"Companion diagnostic assays for PD-1\/PD-L1 checkpoint inhibitors in NSCLC","volume":"16","author":"J\u00f8rgensen","year":"2016","journal-title":"Expert Rev Mol Diagn"},{"key":"2025111813571029800_ocaf142-B15","doi-asserted-by":"crossref","first-page":"757","DOI":"10.1109\/TMI.2020.3021387","article-title":"Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis","volume":"41","author":"Chen","year":"2022","journal-title":"IEEE Trans Med Imaging"},{"key":"2025111813571029800_ocaf142-B16","doi-asserted-by":"crossref","first-page":"i446","DOI":"10.1093\/bioinformatics\/btz342","article-title":"Deep learning with multimodal representation for pancancer prognosis prediction","volume":"35","author":"Cheerla","year":"2019","journal-title":"Bioinformatics"},{"key":"2025111813571029800_ocaf142-B17","first-page":"689","article-title":"Multimodal deep learning","author":"Ngiam","year":"2011","journal-title":"ICML"},{"key":"2025111813571029800_ocaf142-B18","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1162\/neco_a_01273","article-title":"A survey on deep learning for multimodal data fusion","volume":"32","author":"Gao","year":"2020","journal-title":"Neural Comput"},{"key":"2025111813571029800_ocaf142-B19","first-page":"253","volume-title":"Medical Imaging 2017: Biomedical Applications in Molecular, Structural, and Functional Imaging","author":"Paul","year":"2017"},{"key":"2025111813571029800_ocaf142-B20","doi-asserted-by":"crossref","first-page":"2275","DOI":"10.3906\/elk-1801-8","article-title":"Deep learning based brain tumor classification and detection system","volume":"26","author":"Ari","year":"2018","journal-title":"Turkish J Electr Eng Comput Sci"},{"key":"2025111813571029800_ocaf142-B21","doi-asserted-by":"crossref","first-page":"153","DOI":"10.3390\/healthcare9020153","article-title":"A deep learning approach for brain tumor classification and segmentation using a multiscale convolutional neural network","volume":"9","author":"D\u00edaz-Pernas","year":"2021","journal-title":"Healthcare MDPI"},{"key":"2025111813571029800_ocaf142-B22","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.1007\/s10278-024-01001-4","article-title":"Fast real-time brain tumor detection based on stimulated raman histology and self-supervised deep learning model","volume":"37","author":"Wang","year":"2024","journal-title":"J Imaging Inform Med"},{"key":"2025111813571029800_ocaf142-B23","doi-asserted-by":"crossref","first-page":"699","DOI":"10.1109\/EMBC.2015.7318458","volume-title":"2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"Pan","year":"2015"},{"key":"2025111813571029800_ocaf142-B24","doi-asserted-by":"crossref","first-page":"103758","DOI":"10.1016\/j.compbiomed.2020.103758","article-title":"Brain tumor segmentation and grading of lower-grade glioma using deep learning in MRI images","volume":"121","author":"Naser","year":"2020","journal-title":"Comput Biol Med"},{"key":"2025111813571029800_ocaf142-B25","doi-asserted-by":"crossref","first-page":"106047","DOI":"10.1016\/j.cmpb.2021.106047","article-title":"Deep learning-based tumor microenvironment analysis in colon adenocarcinoma histopathological whole-slide images","volume":"204","author":"Jiao","year":"2021","journal-title":"Comput Methods Programs Biomed"},{"key":"2025111813571029800_ocaf142-B26","doi-asserted-by":"crossref","first-page":"1435","DOI":"10.1038\/s41551-022-00951-w","article-title":"Graph deep learning for the characterization of tumour microenvironments from spatial protein profiles in tissue specimens","volume":"6","author":"Wu","year":"2022","journal-title":"Nat Biomed Eng"},{"key":"2025111813571029800_ocaf142-B27","doi-asserted-by":"publisher","first-page":"578","DOI":"10.1002\/path.6153","article-title":"Application of digital pathology-based advanced analytics of tumour microenvironment organisation to predict prognosis and therapeutic response","volume":"260","author":"Fu","year":"2023","journal-title":"J Pathol"},{"key":"2025111813571029800_ocaf142-B28","doi-asserted-by":"crossref","first-page":"6994","DOI":"10.1038\/s41598-019-43372-7","article-title":"Deep learning-based survival prediction of oral cancer patients","volume":"9","author":"Kim","year":"2019","journal-title":"Sci Rep"},{"key":"2025111813571029800_ocaf142-B29","doi-asserted-by":"crossref","first-page":"e0233678","DOI":"10.1371\/journal.pone.0233678","article-title":"Deep learning-based survival prediction for multiple cancer types using histopathology images","volume":"15","author":"Wulczyn","year":"2020","journal-title":"PLoS One"},{"key":"2025111813571029800_ocaf142-B30","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1038\/s41746-021-00427-2","article-title":"Interpretable survival prediction for colorectal cancer using deep learning","volume":"4","author":"Wulczyn","year":"2021","journal-title":"NPJ Digit Med"},{"key":"2025111813571029800_ocaf142-B31","first-page":"1","article-title":"Self-supervised learning: generative or contrastive","volume":"35","author":"Liu","year":"2021","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2025111813571029800_ocaf142-B32","first-page":"16000","author":"He","year":"2022"},{"key":"2025111813571029800_ocaf142-B33","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1093\/nsr\/nwx106","article-title":"A brief introduction to weakly supervised learning","volume":"5","author":"Zhou","year":"2018","journal-title":"Natl Sci Rev"},{"key":"2025111813571029800_ocaf142-B34","first-page":"14866","article-title":"Generating diverse high-fidelity images with vq-vae-2","volume":"32","author":"Razavi","year":"2019","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025111813571029800_ocaf142-B35","doi-asserted-by":"publisher","first-page":"21.4.1","DOI":"10.1002\/0471142735.im2104s103","article-title":"Immunohistochemistry","volume":"103","author":"Hofman","year":"2013","journal-title":"Curr Protoc Immunol"},{"key":"2025111813571029800_ocaf142-B36","first-page":"1725","author":"Guo","year":"2017"},{"key":"2025111813571029800_ocaf142-B37","doi-asserted-by":"crossref","first-page":"1962","DOI":"10.1109\/TMI.2016.2529665","article-title":"Structure-preserving color normalization and sparse stain separation for histological images","volume":"35","author":"Vahadane","year":"2016","journal-title":"IEEE Trans Med Imaging"},{"key":"2025111813571029800_ocaf142-B38","doi-asserted-by":"publisher","first-page":"pl1","DOI":"10.1126\/scisignal.2004088","article-title":"Integrative analysis of complex cancer genomics and clinical profiles using the cBioPortal","volume":"6","author":"Gao","year":"2013","journal-title":"Sci Signal"},{"key":"2025111813571029800_ocaf142-B39","doi-asserted-by":"crossref","first-page":"3861","DOI":"10.1158\/0008-5472.CAN-23-0816","article-title":"Analysis and visualization of longitudinal genomic and clinical data from the AACR project GENIE biopharma collaborative in cBioPortal","volume":"83","author":"de Bruijn","year":"2023","journal-title":"Cancer Res"},{"key":"2025111813571029800_ocaf142-B40","first-page":"12031","author":"Bhattacharjee","year":"2022"},{"key":"2025111813571029800_ocaf142-B41","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1038\/s41591-020-1131-x","article-title":"Liver metastasis restrains immunotherapy efficacy via macrophage-mediated T cell elimination","volume":"27","author":"Yu","year":"2021","journal-title":"Nat Med"},{"key":"2025111813571029800_ocaf142-B42","doi-asserted-by":"crossref","first-page":"829451","DOI":"10.3389\/fimmu.2022.829451","article-title":"Tumor immune microenvironment and immunotherapy in brain metastasis from non-small cell lung cancer","volume":"13","author":"Wang","year":"2022","journal-title":"Front Immunol"},{"key":"2025111813571029800_ocaf142-B43","doi-asserted-by":"crossref","first-page":"e002230","DOI":"10.1136\/jitc-2020-002230","article-title":"Comparison of programmed death-ligand 1 protein expression between primary and metastatic lesions in patients with lung cancer","volume":"9","author":"Moutafi","year":"2021","journal-title":"J Immunother Cancer"},{"key":"2025111813571029800_ocaf142-B44","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.ijrobp.2020.04.018","article-title":"Programmed death receptor ligand one expression may independently predict survival in patients with non-small cell lung carcinoma brain metastases receiving immunotherapy","volume":"108","author":"Hulsbergen","year":"2020","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"2025111813571029800_ocaf142-B45","doi-asserted-by":"crossref","first-page":"254","DOI":"10.6004\/jnccn.2021.0013","article-title":"NCCN guidelines insights: non-small cell lung cancer, version 2.2021: featured updates to the NCCN guidelines","volume":"19","author":"Ettinger","year":"2021","journal-title":"J Natl Comprehensive Cancer Netw"},{"key":"2025111813571029800_ocaf142-B46","doi-asserted-by":"crossref","first-page":"100260","DOI":"10.1016\/j.esmoop.2021.100260","article-title":"Targeting HER2 in non-small-cell lung cancer (NSCLC): a glimpse of hope? An updated review on therapeutic strategies in NSCLC harbouring HER2 alterations","volume":"6","author":"Riudavets","year":"2021","journal-title":"ESMO Open"},{"key":"2025111813571029800_ocaf142-B47","doi-asserted-by":"crossref","first-page":"1321","DOI":"10.1093\/annonc\/mdz167","article-title":"Immune checkpoint inhibitors for patients with advanced lung cancer and oncogenic driver alterations: results from the IMMUNOTARGET registry","volume":"30","author":"Mazieres","year":"2019","journal-title":"Ann Oncol"},{"key":"2025111813571029800_ocaf142-B48","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1016\/j.jtho.2019.12.129","article-title":"Efficacy and safety of anti-PD-1 immunotherapy in patients with advanced NSCLC with BRAF, HER2, or MET mutations or RET translocation: GFPC 01-2018","volume":"15","author":"Guisier","year":"2020","journal-title":"J Thorac Oncol"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/11\/1641\/64200484\/ocaf142.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/11\/1641\/64200484\/ocaf142.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T18:57:22Z","timestamp":1763492242000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/11\/1641\/8246850"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,3]]},"references-count":48,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,9,3]]},"published-print":{"date-parts":[[2025,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf142","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,11]]},"published":{"date-parts":[[2025,9,3]]}}}