{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T16:10:49Z","timestamp":1779379849419,"version":"3.53.1"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T00:00:00Z","timestamp":1701129600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T00:00:00Z","timestamp":1701129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013873","name":"Government of Ontario","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100013873","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004381","name":"Western University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004381","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100019711","name":"London Health Sciences Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100019711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Qualitative observer-based and quantitative radiomics-based analyses of T1w contrast-enhanced magnetic resonance imaging (T1w-CE MRI) have both been shown to predict the outcomes of brain metastasis (BM) stereotactic radiosurgery (SRS). Comparison of these methods and interpretation of radiomics-based machine learning (ML) models remains limited. To address this need, we collected a dataset of n\u2009=\u2009123 BMs from 99 patients including 12 clinical features, 107 pre-treatment T1w-CE MRI radiomic features, and BM post-SRS progression scores. A previously published outcome model using SRS dose prescription and five-way BM qualitative appearance scoring was evaluated. We found high qualitative scoring interobserver variability across five observers that negatively impacted the model\u2019s risk stratification. Radiomics-based ML models trained to replicate the qualitative scoring did so with high accuracy (bootstrap-corrected AUC\u2009=\u20090.84\u20130.94), but risk stratification using these replicated qualitative scores remained poor. Radiomics-based ML models trained to directly predict post-SRS progression offered enhanced risk stratification (Kaplan\u2013Meier rank-sum <jats:italic>p<\/jats:italic>\u2009=\u20090.0003) compared to using qualitative appearance. The qualitative appearance scoring enabled interpretation of the progression radiomics-based ML model, with necrotic BMs and a subset of heterogeneous BMs predicted as being at high-risk of post-SRS progression, in agreement with current radiobiological understanding. Our study\u2019s results show that while radiomics-based SRS outcome models out-perform qualitative appearance analysis, qualitative appearance still provides critical insight into ML model operation.<\/jats:p>","DOI":"10.1038\/s41598-023-47702-8","type":"journal-article","created":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T14:02:47Z","timestamp":1701180167000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Predicting stereotactic radiosurgery outcomes with multi-observer qualitative appearance labelling versus MRI radiomics"],"prefix":"10.1038","volume":"13","author":[{"given":"David A.","family":"DeVries","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Terence","family":"Tang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Albweady","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Leung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joanna","family":"Laba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carol","family":"Johnson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank","family":"Lagerwaard","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaap","family":"Zindler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George","family":"Hajdok","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aaron D.","family":"Ward","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,11,28]]},"reference":[{"key":"47702_CR1","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1007\/s11912-011-0203-y","volume":"14","author":"L Nayak","year":"2012","unstructured":"Nayak, L., Lee, E. Q. & Wen, P. Y. Epidemiology of brain metastases. Curr. Oncol. Rep. 14, 48\u201354 (2012).","journal-title":"Curr. Oncol. Rep."},{"key":"47702_CR2","doi-asserted-by":"publisher","first-page":"3773","DOI":"10.1200\/JCO.20.01255","volume":"38","author":"PW Sperduto","year":"2020","unstructured":"Sperduto, P. W. et al. Survival in patients with brain metastases: Summary report on the updated diagnosis-specific graded prognostic assessment and definition of the eligibility quotient. J. Clin. Oncol. 38, 3773\u20133784 (2020).","journal-title":"J. Clin. Oncol."},{"key":"47702_CR3","doi-asserted-by":"publisher","first-page":"1332","DOI":"10.1016\/j.annonc.2021.07.016","volume":"32","author":"E Le Rhun","year":"2021","unstructured":"Le Rhun, E. et al. EANO\u2013ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up of patients with brain metastasis from solid tumours. Ann. Oncol. 32, 1332\u20131347 (2021).","journal-title":"Ann. Oncol."},{"key":"47702_CR4","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1093\/neuonc\/noab262","volume":"24","author":"MA Vogelbaum","year":"2022","unstructured":"Vogelbaum, M. A. et al. Treatment for brain metastases: ASCO-SNO-ASTRO guideline. Neuro. Oncol. 24, 331\u2013357 (2022).","journal-title":"Neuro. Oncol."},{"key":"47702_CR5","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1093\/neuros\/nyx522","volume":"83","author":"ST Chao","year":"2018","unstructured":"Chao, S. T. et al. Stereotactic radiosurgery in the management of limited (1\u20134) brain metasteses: Systematic review and International Stereotactic Radiosurgery Society practice guideline. Neurosurgery 83, 345\u2013353 (2018).","journal-title":"Neurosurgery"},{"key":"47702_CR6","doi-asserted-by":"publisher","first-page":"373","DOI":"10.3171\/2014.10.JNS141610","volume":"123","author":"PK Sneed","year":"2015","unstructured":"Sneed, P. K. et al. Adverse radiation effect after stereotactic radiosurgery for brain metastases: Incidence, time course, and risk factors. J. Neurosurg. 123, 373\u2013386 (2015).","journal-title":"J. Neurosurg."},{"key":"47702_CR7","doi-asserted-by":"publisher","first-page":"996","DOI":"10.1016\/j.ijrobp.2009.06.006","volume":"77","author":"BJ Blonigen","year":"2010","unstructured":"Blonigen, B. J. et al. Irradiated volume as a predictor of brain radionecrosis after linear accelerator stereotactic radiosurgery. Int. J. Radiat. Oncol. 77, 996\u20131001 (2010).","journal-title":"Int. J. Radiat. Oncol."},{"key":"47702_CR8","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1016\/S0360-3016(97)00154-5","volume":"39","author":"C Nieder","year":"1997","unstructured":"Nieder, C., Berberich, W. & Schnabel, K. Tumor-related prognostic factors for remission of brain metastases after radiotherapy. Int. J. Radiat. Oncol. Biol. Phys. 39, 25\u201330 (1997).","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"47702_CR9","doi-asserted-by":"publisher","first-page":"2075","DOI":"10.1002\/(SICI)1097-0142(19971201)80:11<2075::AID-CNCR6>3.0.CO;2-W","volume":"80","author":"YS Kim","year":"1997","unstructured":"Kim, Y. S., Kondziolka, D., Flickinger, J. C. & Lunsford, L. D. Stereotactic radiosurgery for patients with nonsmall cell lung carcinoma metastatic to the brain. Cancer 80, 2075\u20132083 (1997).","journal-title":"Cancer"},{"key":"47702_CR10","doi-asserted-by":"publisher","first-page":"807","DOI":"10.1148\/radiology.211.3.r99jn48807","volume":"211","author":"AM Peterson","year":"1999","unstructured":"Peterson, A. M., Meltzer, C. C., Evanson, E. J., Flickinger, J. C. & Kondziolka, D. MR imaging response of brain metastases after Gamma Knife stereotactic radiosurgery. Radiology 211, 807\u2013814 (1999).","journal-title":"Radiology"},{"key":"47702_CR11","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1016\/S0360-3016(00)01584-4","volume":"50","author":"KA Goodman","year":"2001","unstructured":"Goodman, K. A. et al. Relationship between pattern of enhancement and local control of brain metastases after radiosurgery. Int. J. Radiat. Oncol. Biol. Phys. 50, 139\u2013146 (2001).","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"47702_CR12","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1016\/S0360-3016(96)00497-X","volume":"37","author":"C-Y Shiau","year":"1997","unstructured":"Shiau, C.-Y. et al. Radiosurgery for brain metastases: Relationship of dose and pattern of enhancement to local control. Int. J. Radiat. Oncol. Biol. Phys. 37, 375\u2013383 (1997).","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"47702_CR13","first-page":"311","volume":"17","author":"Y Naoi","year":"1999","unstructured":"Naoi, Y., Maehara, T., Cho, N. & Katayama, H. Stereotactic radiosurgery for brain metastases using a linac system: Evaluation of initial local response by imaging. Radiat. Med. 17, 311\u2013315 (1999).","journal-title":"Radiat. Med."},{"key":"47702_CR14","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1634\/theoncologist.2012-0316","volume":"18","author":"G Rodrigues","year":"2013","unstructured":"Rodrigues, G., Zindler, J., Warner, A. & Lagerwaard, F. Recursive partitioning analysis for the prediction of stereotactic radiosurgery brain metastases lesion control. Oncologist 18, 330\u2013335 (2013).","journal-title":"Oncologist"},{"key":"47702_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-022-25389-7","volume":"12","author":"DA DeVries","year":"2022","unstructured":"DeVries, D. A. et al. Performance sensitivity analysis of brain metastasis stereotactic radiosurgery outcome prediction using MRI radiomics. Sci. Rep. 12, 1\u201312 (2022).","journal-title":"Sci. Rep."},{"key":"47702_CR16","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.radonc.2021.11.010","volume":"166","author":"R Gutsche","year":"2022","unstructured":"Gutsche, R. et al. Radiomics outperforms semantic features for prediction of response to stereotactic radiosurgery in brain metastases. Radiother. Oncol. 166, 37\u201343 (2022).","journal-title":"Radiother. Oncol."},{"key":"47702_CR17","doi-asserted-by":"publisher","first-page":"2266","DOI":"10.1007\/s00330-021-08368-w","volume":"32","author":"ZK Jiang","year":"2022","unstructured":"Jiang, Z. K. et al. Multimodality MRI-based radiomics approach to predict the posttreatment response of lung cancer brain metastases to gamma knife radiosurgery. Eur. Radiol. 32, 2266\u20132276 (2022).","journal-title":"Eur. Radiol."},{"key":"47702_CR18","doi-asserted-by":"publisher","first-page":"569461","DOI":"10.3389\/fonc.2020.569461","volume":"10","author":"D Kawahara","year":"2021","unstructured":"Kawahara, D., Tang, X., Lee, C. K., Nagata, Y. & Watanabe, Y. Predicting the local response of metastatic brain tumor to Gamma Knife radiosurgery by radiomics with a machine learning method. Front. Oncol. 10, 569461 (2021).","journal-title":"Front. Oncol."},{"key":"47702_CR19","doi-asserted-by":"publisher","first-page":"4030","DOI":"10.3390\/cancers13164030","volume":"13","author":"CY Liao","year":"2021","unstructured":"Liao, C. Y. et al. Enhancement of radiosurgical treatment outcome prediction using MRI radiomics in patients with non-small cell lung cancer brain metastases. Cancers 13, 4030 (2021).","journal-title":"Cancers"},{"key":"47702_CR20","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1093\/neuonc\/noaa007","volume":"22","author":"A Mouraviev","year":"2020","unstructured":"Mouraviev, A. et al. Use of radiomics for the prediction of local control of brain metastases after stereotactic radiosurgery. Neuro Oncol. 22, 797\u2013805 (2020).","journal-title":"Neuro Oncol."},{"key":"47702_CR21","doi-asserted-by":"publisher","first-page":"5522","DOI":"10.1002\/mp.15110","volume":"48","author":"HS Wang","year":"2021","unstructured":"Wang, H. S. et al. Predicting local failure of brain metastases after stereotactic radiosurgery with radiomics on planning MR images and dose maps. Med. Phys. 48, 5522\u20135530 (2021).","journal-title":"Med. Phys."},{"key":"47702_CR22","doi-asserted-by":"publisher","first-page":"1114194","DOI":"10.3389\/fonc.2023.1114194","volume":"13","author":"P Du","year":"2023","unstructured":"Du, P. et al. Prediction of treatment response in patients with brain metastasis receiving stereotactic radiosurgery based on pre-treatment multimodal MRI radiomics and clinical risk factors: A machine learning model. Front. Oncol. 13, 1114194 (2023).","journal-title":"Front. Oncol."},{"key":"47702_CR23","doi-asserted-by":"publisher","first-page":"109424","DOI":"10.1016\/j.radonc.2022.11.013","volume":"178","author":"G Carloni","year":"2023","unstructured":"Carloni, G. et al. Brain metastases from NSCLC treated with stereotactic radiotherapy: Prediction mismatch between two different radiomic platforms. Radiother. Oncol. 178, 109424 (2023).","journal-title":"Radiother. Oncol."},{"key":"47702_CR24","first-page":"27","volume":"29","author":"K Mulford","year":"2021","unstructured":"Mulford, K. et al. A radiomics-based model for predicting local control of resected brain metastases receiving adjuvant SRS. Clin. Transl. Radiat. Oncol. 29, 27\u201332 (2021).","journal-title":"Clin. Transl. Radiat. Oncol."},{"key":"47702_CR25","doi-asserted-by":"publisher","first-page":"21620","DOI":"10.1038\/s41598-021-01024-9","volume":"11","author":"M Jaberipour","year":"2021","unstructured":"Jaberipour, M., Soliman, H., Sahgal, A. & Sadeghi-Naini, A. A priori prediction of local failure in brain metastasis after hypo-fractionated stereotactic radiotherapy using quantitative MRI and machine learning. Sci. Rep. 11, 21620 (2021).","journal-title":"Sci. Rep."},{"key":"47702_CR26","doi-asserted-by":"publisher","first-page":"7167","DOI":"10.1002\/mp.15814","volume":"49","author":"SA Jalalifar","year":"2022","unstructured":"Jalalifar, S. A., Soliman, H., Sahgal, A. & Sadeghi-Naini, A. Predicting the outcome of radiotherapy in brain metastasis by integrating the clinical and MRI-based deep learning features. Med. Phys. 49, 7167\u20137178 (2022).","journal-title":"Med. Phys."},{"key":"47702_CR27","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1109\/JTEHM.2022.3219625","volume":"11","author":"SA Jalalifar","year":"2022","unstructured":"Jalalifar, S. A., Soliman, H., Sahgal, A. & Sadeghi-Naini, A. A self-attention-guided 3D deep residual network with big transfer to predict local failure in brain metastasis after radiotherapy using multi-channel MRI. IEEE J. Transl. Eng. Heal. Med. 11, 13\u201322 (2022).","journal-title":"IEEE J. Transl. Eng. Heal. Med."},{"key":"47702_CR28","doi-asserted-by":"publisher","first-page":"19830","DOI":"10.1038\/s41598-019-56185-5","volume":"9","author":"E Karami","year":"2019","unstructured":"Karami, E. et al. Quantitative MRI biomarkers of stereotactic radiotherapy outcome in brain metastasis. Sci. Rep. 9, 19830 (2019).","journal-title":"Sci. Rep."},{"key":"47702_CR29","doi-asserted-by":"publisher","first-page":"e104","DOI":"10.1158\/0008-5472.CAN-17-0339","volume":"77","author":"JJM van Griethuysen","year":"2017","unstructured":"van Griethuysen, J. J. M. et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77, e104\u2013e107 (2017).","journal-title":"Cancer Res."},{"key":"47702_CR30","doi-asserted-by":"publisher","first-page":"563","DOI":"10.1148\/radiol.2015151169","volume":"278","author":"RJ Gillies","year":"2015","unstructured":"Gillies, R. J., Kinahan, P. E. & Hricak, H. Radiomics: Images are more than pictures, they are data. Radiology 278, 563\u2013577 (2015).","journal-title":"Radiology"},{"key":"47702_CR31","doi-asserted-by":"publisher","first-page":"e270","DOI":"10.1016\/S1470-2045(15)70057-4","volume":"16","author":"NU Lin","year":"2015","unstructured":"Lin, N. U. et al. Response assessment criteria for brain metastases: Proposal from the RANO group. Lancet Oncol. 16, e270\u2013e278 (2015).","journal-title":"Lancet Oncol."},{"key":"47702_CR32","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/j.mri.2003.09.001","volume":"22","author":"G Collewet","year":"2004","unstructured":"Collewet, G., Strzelecki, M. & Mariette, F. Influence of MRI acquisition protocols and image intensity normalization methods on texture classification. Magn. Reson. Imaging 22, 81\u201391 (2004).","journal-title":"Magn. Reson. Imaging"},{"key":"47702_CR33","first-page":"548","volume":"92","author":"B Efron","year":"1997","unstructured":"Efron, B. & Tibshirani, R. Improvements on cross-validation: The .632+ bootstrap method. J. Am. Stat. Assoc. 92, 548\u2013560 (1997).","journal-title":"J. Am. Stat. Assoc."},{"key":"47702_CR34","doi-asserted-by":"publisher","first-page":"1323","DOI":"10.1016\/j.mri.2012.05.001","volume":"30","author":"A Fedorov","year":"2012","unstructured":"Fedorov, A. et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging 30, 1323\u20131341 (2012).","journal-title":"Magn. Reson. Imaging"},{"key":"47702_CR35","unstructured":"Molnar, C. Interpretable Machine Learning. https:\/\/christophm.github.io\/interpretable-ml-book (2022)."},{"key":"47702_CR36","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1111\/rssb.12377","volume":"82","author":"DW Apley","year":"2020","unstructured":"Apley, D. W. & Zhu, J. Visualizing the effects of predictor variables in black box supervised learning models. J. R. Stat. Soc. Ser. B Stat. Methodol. 82, 1059\u20131086 (2020).","journal-title":"J. R. Stat. Soc. Ser. B Stat. Methodol."},{"key":"47702_CR37","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1038\/nrclinonc.2012.171","volume":"9","author":"MR Horsman","year":"2012","unstructured":"Horsman, M. R., Mortensen, L. S., Petersen, J. B., Busk, M. & Overgaard, J. Imaging hypoxia to improve radiotherapy outcome. Nat. Rev. Clin. Oncol. 9, 674\u2013687 (2012).","journal-title":"Nat. Rev. Clin. Oncol."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-47702-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-47702-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-47702-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,28]],"date-time":"2023-11-28T14:12:07Z","timestamp":1701180727000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-47702-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,28]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["47702"],"URL":"https:\/\/doi.org\/10.1038\/s41598-023-47702-8","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,28]]},"assertion":[{"value":"10 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"20977"}}