{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T01:36:51Z","timestamp":1769909811761,"version":"3.49.0"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T00:00:00Z","timestamp":1697587200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T00:00:00Z","timestamp":1697587200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"The Norwegian Cancer Society and the Research Council of Norway","award":["Grants agreements No 303249, 261984, 325971"],"award-info":[{"award-number":["Grants agreements No 303249, 261984, 325971"]}]},{"name":"The Norwegian Cancer Society and the Research Council of Norway","award":["Grants agreements No 303249, 261984, 325971"],"award-info":[{"award-number":["Grants agreements No 303249, 261984, 325971"]}]},{"name":"The Norwegian Cancer Society and the Research Council of Norway","award":["Grants agreements No 303249, 261984, 325971"],"award-info":[{"award-number":["Grants agreements No 303249, 261984, 325971"]}]},{"DOI":"10.13039\/501100006095","name":"South-Eastern Norway Regional Health Authority","doi-asserted-by":"crossref","award":["Grants agreements No 2021057, 2013069, 2017073"],"award-info":[{"award-number":["Grants agreements No 2021057, 2013069, 2017073"]}],"id":[{"id":"10.13039\/501100006095","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100006095","name":"South-Eastern Norway Regional Health Authority","doi-asserted-by":"crossref","award":["Grants agreements No 2021057, 2013069, 2017073"],"award-info":[{"award-number":["Grants agreements No 2021057, 2013069, 2017073"]}],"id":[{"id":"10.13039\/501100006095","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100006095","name":"South-Eastern Norway Regional Health Authority","doi-asserted-by":"crossref","award":["Grants agreements No 2021057, 2013069, 2017073"],"award-info":[{"award-number":["Grants agreements No 2021057, 2013069, 2017073"]}],"id":[{"id":"10.13039\/501100006095","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Marie Sk\u0142odowska-Curie","award":["grant agreement No 844646-GLIOHAB"],"award-info":[{"award-number":["grant agreement No 844646-GLIOHAB"]}]},{"name":"Marie Sk\u0142odowska-Curie","award":["grant agreement No 844646-GLIOHAB"],"award-info":[{"award-number":["grant agreement No 844646-GLIOHAB"]}]},{"name":"Spanish State Research Agency, Subprogram for Knowledge Generation - PROGRESS","award":["Grant agreement No PID2021-127110OA-I00"],"award-info":[{"award-number":["Grant agreement No PID2021-127110OA-I00"]}]},{"name":"The European Union\u2019s Horizon 2020 Programmes","award":["ERC Grant Agreement No. 758657-ImPRESS"],"award-info":[{"award-number":["ERC Grant Agreement No. 758657-ImPRESS"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Saliency-based algorithms are able to explain the relationship between input image pixels and deep-learning model predictions. However, it may be difficult to assess the clinical value of the most important image features and the model predictions derived from the raw saliency map. This study proposes to enhance the interpretability of saliency-based deep learning model for survival classification of patients with gliomas, by extracting domain knowledge-based information from the raw saliency maps.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Materials and methods<\/jats:title>\n                <jats:p>Our study includes presurgical T1-weighted (pre- and post-contrast), T2-weighted and T2-FLAIR MRIs of 147 glioma patients from the BraTs 2020 challenge dataset aligned to the SRI 24 anatomical atlas. Each image exam includes a segmentation mask and the information of overall survival (OS) from time of diagnosis (in days). This dataset was divided into training (<jats:inline-formula><jats:alternatives><jats:tex-math>$$n=118$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mi>n<\/mml:mi>\n                      <mml:mo>=<\/mml:mo>\n                      <mml:mn>118<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula>) and validation (<jats:inline-formula><jats:alternatives><jats:tex-math>$$n=29$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mi>n<\/mml:mi>\n                      <mml:mo>=<\/mml:mo>\n                      <mml:mn>29<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula>) datasets. The extent of surgical resection for all patients was gross total resection. We categorized the data into 42 short (mean <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mu =157$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mi>\u03bc<\/mml:mi>\n                      <mml:mo>=<\/mml:mo>\n                      <mml:mn>157<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> days), 30 medium (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mu =369$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mi>\u03bc<\/mml:mi>\n                      <mml:mo>=<\/mml:mo>\n                      <mml:mn>369<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> days), and 46 long (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\mu =761$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mi>\u03bc<\/mml:mi>\n                      <mml:mo>=<\/mml:mo>\n                      <mml:mn>761<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> days) survivors. A 3D convolutional neural network (CNN) trained on brain tumour MRI volumes classified all patients based on expected prognosis of either short-term, medium-term, or long-term survival. We extend the popular 2D Gradient-weighted Class Activation Mapping (Grad-CAM), for the generation of saliency map, to 3D and combined it with the anatomical atlas, to extract brain regions, brain volume and probability map that reveal domain knowledge-based information.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>For each OS class, a larger tumor volume was associated with a shorter OS. There were 10, 7 and 27 tumor locations in brain regions that uniquely associate with the short-term, medium-term, and long-term survival, respectively. Tumors located in the transverse temporal gyrus, fusiform, and palladium are associated with short, medium and long-term survival, respectively. The visual and textual information displayed during OS prediction highlights tumor location and the contribution of different brain regions to the prediction of OS. This algorithm design feature assists the physician in analyzing and understanding different model prediction stages.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Domain knowledge-based information extracted from the saliency map can enhance the interpretability of deep learning models. Our findings show that tumors overlapping eloquent brain regions are associated with short patient survival.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-023-02320-2","type":"journal-article","created":{"date-parts":[[2023,10,18]],"date-time":"2023-10-18T11:01:57Z","timestamp":1697626917000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Interpreting deep learning models for glioma survival classification using visualization and textual explanations"],"prefix":"10.1186","volume":"23","author":[{"given":"Michael","family":"Osadebey","sequence":"first","affiliation":[]},{"given":"Qinghui","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Elies","family":"Fuster-Garcia","sequence":"additional","affiliation":[]},{"given":"Kyrre E.","family":"Emblem","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,10,18]]},"reference":[{"issue":"8","key":"2320_CR1","doi-asserted-by":"publisher","first-page":"1231","DOI":"10.1093\/neuonc\/noab106","volume":"23","author":"DN Louis","year":"2021","unstructured":"Louis DN, Perry A, Wesseling P, Brat DJ, Cree IA, Figarella-Branger D, et al. The 2021 WHO classification of tumors of the central nervous system: a summary. Neuro-Oncol. 2021;23(8):1231\u201351.","journal-title":"Neuro-Oncol."},{"key":"2320_CR2","doi-asserted-by":"crossref","unstructured":"Ostrom QT, Gittleman H, Xu J, Kromer C, Wolinsky Y, Kruchko C, et\u00a0al. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2009\u20132013. Neuro-Oncol. 2016;18(suppl_5):1\u201375.","DOI":"10.1093\/neuonc\/now207"},{"key":"2320_CR3","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/j.clineuro.2017.11.016","volume":"164","author":"CK Patel","year":"2018","unstructured":"Patel CK, Vemaraju R, Glasbey J, Shires J, Northmore T, Zaben M, et al. Trends in peri-operative performance status following resection of high grade glioma and brain metastases: the impact on survival. Clin Neurol Neurosurg. 2018;164:67\u201371.","journal-title":"Clin Neurol Neurosurg."},{"issue":"1","key":"2320_CR4","doi-asserted-by":"publisher","first-page":"271","DOI":"10.20892\/j.issn.2095-3941.2020.0179","volume":"18","author":"R Chai","year":"2021","unstructured":"Chai R, Li G, Liu Y, Zhang K, Zhao Z, Wu F, et al. Predictive value of MGMT promoter methylation on the survival of TMZ treated IDH-mutant glioblastoma. Cancer Biol Med. 2021;18(1):271.","journal-title":"Cancer Biol Med."},{"issue":"9","key":"2320_CR5","doi-asserted-by":"publisher","first-page":"2030","DOI":"10.3390\/jcm10092030","volume":"10","author":"FY Chiu","year":"2021","unstructured":"Chiu FY, Le NQK, Chen CY. A multiparametric MRI-based radiomics analysis to efficiently classify tumor subregions of glioblastoma: A pilot study in machine learning. J Clin Med. 2021;10(9):2030.","journal-title":"J Clin Med."},{"issue":"12","key":"2320_CR6","doi-asserted-by":"publisher","first-page":"3067","DOI":"10.1007\/s00701-020-04483-7","volume":"162","author":"Y Wan","year":"2020","unstructured":"Wan Y, Rahmat R, Price SJ. Deep learning for glioblastoma segmentation using preoperative magnetic resonance imaging identifies volumetric features associated with survival. Acta Neurochir. 2020;162(12):3067\u201380.","journal-title":"Acta Neurochir."},{"issue":"1","key":"2320_CR7","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","volume":"25","author":"EJ Topol","year":"2019","unstructured":"Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44\u201356.","journal-title":"Nat Med."},{"key":"2320_CR8","doi-asserted-by":"crossref","unstructured":"Li Y, Shen L. Deep learning based multimodal brain tumor diagnosis. In: International MICCAI Brainlesion Workshop. Springer; 2017. p. 149\u2013158.","DOI":"10.1007\/978-3-319-75238-9_13"},{"key":"2320_CR9","doi-asserted-by":"crossref","unstructured":"Suter Y, Jungo A, Rebsamen M, Knecht U, Herrmann E, Wiest R, et\u00a0al. Deep learning versus classical regression for brain tumor patient survival prediction. In: International MICCAI Brainlesion Workshop. Springer; 2018. p. 429\u2013440.","DOI":"10.1007\/978-3-030-11726-9_38"},{"key":"2320_CR10","doi-asserted-by":"crossref","unstructured":"Han IS. Multimodal brain image analysis and survival prediction using neuromorphic attention-based neural networks. In: International MICCAI Brainlesion Workshop. Springer; 2020. p. 194\u2013206.","DOI":"10.1007\/978-3-030-72084-1_18"},{"key":"2320_CR11","unstructured":"Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What clinicians want: contextualizing explainable machine learning for clinical end use. In: Machine learning for healthcare conference. PMLR; 2019. p. 359\u2013380."},{"key":"2320_CR12","doi-asserted-by":"crossref","unstructured":"Reyes M, Meier R, Pereira S, Silva CA, Dahlweid FM, Tengg-Kobligk Hv, et\u00a0al. On the interpretability of artificial intelligence in radiology: challenges and opportunities. Radiol Artif Intell. 2020;2(3):190043.","DOI":"10.1148\/ryai.2020190043"},{"key":"2320_CR13","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1109\/RBME.2021.3131358","volume":"16","author":"JP Amorim","year":"2021","unstructured":"Amorim JP, Abreu PH, Fern\u00e1ndez A, Reyes M, Santos J, Abreu MH. Interpreting deep machine learning models: an easy guide for oncologists. IEEE Rev Biomed Eng. 2021;16:192\u2013206.","journal-title":"IEEE Rev Biomed Eng"},{"key":"2320_CR14","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. New York: Springer-nature; 2017. p. 618\u201326.","DOI":"10.1109\/ICCV.2017.74"},{"key":"2320_CR15","unstructured":"Smilkov D, Thorat N, Kim B, Vi\u00e9gas F, Wattenberg M. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825. 2017."},{"issue":"14","key":"2320_CR16","doi-asserted-by":"publisher","first-page":"11347","DOI":"10.1007\/s00521-020-05624-w","volume":"34","author":"C Ieracitano","year":"2022","unstructured":"Ieracitano C, Mammone N, Hussain A, Morabito FC. A novel explainable machine learning approach for EEG-based brain-computer interface systems. Neural Comput & Applic. 2022;34(14):11347\u201360.","journal-title":"Neural Comput & Applic."},{"key":"2320_CR17","first-page":"4765","volume":"30","author":"SM Lundberg","year":"2017","unstructured":"Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Adv Neural Inf Process Syst. 2017;30:4765\u201376.","journal-title":"Adv Neural Inf Process Syst"},{"issue":"11","key":"2320_CR18","doi-asserted-by":"publisher","first-page":"2660","DOI":"10.1109\/TNNLS.2016.2599820","volume":"28","author":"W Samek","year":"2016","unstructured":"Samek W, Binder A, Montavon G, Lapuschkin S, M\u00fcller KR. Evaluating the visualization of what a deep neural network has learned. IEEE Trans Neural Netw Learn Syst. 2016;28(11):2660\u201373.","journal-title":"IEEE Trans. Neural Netw Learn Syst."},{"issue":"4","key":"2320_CR19","doi-asserted-by":"publisher","first-page":"1834","DOI":"10.3390\/app12041834","volume":"12","author":"S Chatterjee","year":"2022","unstructured":"Chatterjee S, Das A, Mandal C, Mukhopadhyay B, Vipinraj M, Shukla A, et al. TorchEsegeta: Framework for Interpretability and Explainability of Image-Based Deep Learning Models. Appl Sci. 2022;12(4):1834.","journal-title":"Appl Sci."},{"issue":"6","key":"2320_CR20","doi-asserted-by":"publisher","first-page":"52","DOI":"10.3390\/jimaging6060052","volume":"6","author":"A Singh","year":"2020","unstructured":"Singh A, Sengupta S, Lakshminarayanan V. Explainable deep learning models in medical image analysis. J Imaging. 2020;6(6):52.","journal-title":"J Imaging."},{"issue":"1","key":"2320_CR21","first-page":"1064","volume":"26","author":"T Spinner","year":"2019","unstructured":"Spinner T, Schlegel U, Sch\u00e4fer H, El-Assady M. explAIner: A visual analytics framework for interactive and explainable machine learning. IEEE Trans Vis Comput Graph. 2019;26(1):1064\u201374.","journal-title":"IEEE Trans Vis Comput Graph."},{"issue":"10","key":"2320_CR22","doi-asserted-by":"publisher","first-page":"1993","DOI":"10.1109\/TMI.2014.2377694","volume":"34","author":"BH Menze","year":"2014","unstructured":"Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging. 2014;34(10):1993\u20132024.","journal-title":"IEEE Trans Med Imaging."},{"issue":"5","key":"2320_CR23","doi-asserted-by":"publisher","first-page":"798","DOI":"10.1002\/hbm.20906","volume":"31","author":"T Rohlfing","year":"2010","unstructured":"Rohlfing T, Zahr NM, Sullivan EV, Pfefferbaum A. The SRI24 multichannel atlas of normal adult human brain structure. Hum Brain Mapp. 2010;31(5):798\u2013819.","journal-title":"Hum Brain Mapp."},{"key":"2320_CR24","doi-asserted-by":"crossref","unstructured":"Sethy SS. Introduction to Logic and Logical Discourse. Springer Nature; 2021.","DOI":"10.1007\/978-981-16-2689-0"},{"key":"2320_CR25","doi-asserted-by":"publisher","first-page":"448","DOI":"10.1016\/j.wneu.2018.04.067","volume":"115","author":"S Leu","year":"2018","unstructured":"Leu S, Boulay JL, Thommen S, Bucher HC, Stippich C, Mariani L, et al. Preoperative two-dimensional size of glioblastoma is associated with patient survival. World Neurosurg. 2018;115:448\u201363.","journal-title":"World Neurosurg."},{"issue":"7","key":"2320_CR26","doi-asserted-by":"publisher","first-page":"1895","DOI":"10.1007\/s00701-021-04802-6","volume":"163","author":"EH Fyllingen","year":"2021","unstructured":"Fyllingen EH, B\u00f8 LE, Reinertsen I, Jakola AS, Sagberg LM, Berntsen EM, et al. Survival of glioblastoma in relation to tumor location: a statistical tumor atlas of a population-based cohort. Acta Neurochir. 2021;163(7):1895\u2013905.","journal-title":"Acta Neurochir."},{"issue":"1","key":"2320_CR27","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1016\/j.ijrobp.2019.06.2282","volume":"105","author":"G Chua","year":"2019","unstructured":"Chua G, Chua K, Chua E, Wong F, Kusumawidjaja G. Tumor Location of GBM Predicts for Survival. Int J Radiat Oncol Biol Phys. 2019;105(1):96\u20137.","journal-title":"Int J Radiat Oncol Biol Phys."},{"issue":"3","key":"2320_CR28","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1148\/radiol.2019190491","volume":"293","author":"A Roux","year":"2019","unstructured":"Roux A, Roca P, Edjlali M, Sato K, Zanello M, Dezamis E, et al. MRI atlas of IDH wild-type supratentorial glioblastoma: probabilistic maps of phenotype, management, and outcomes. Radiology. 2019;293(3):633\u201343.","journal-title":"Radiology."},{"issue":"1","key":"2320_CR29","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1093\/neuonc\/now187","volume":"19","author":"BM Ellingson","year":"2017","unstructured":"Ellingson BM, Harris RJ, Woodworth DC, Leu K, Zaw O, Mason WP, et al. Baseline pretreatment contrast enhancing tumor volume including central necrosis is a prognostic factor in recurrent glioblastoma: evidence from single and multicenter trials. Neuro-Oncol. 2017;19(1):89\u201398.","journal-title":"Neuro-Oncol."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-023-02320-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-023-02320-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-023-02320-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T07:13:15Z","timestamp":1700377995000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-023-02320-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,18]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["2320"],"URL":"https:\/\/doi.org\/10.1186\/s12911-023-02320-2","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,18]]},"assertion":[{"value":"12 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 October 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"225"}}