{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,11,20]],"date-time":"2024-11-20T05:11:14Z","timestamp":1732079474309,"version":"3.28.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Vis. Comput. Ind. Biomed. Art"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>To conduct a computational investigation to explore the influence of clinical reference uncertainty on magnetic resonance imaging (MRI) radiomics feature selection, modelling, and performance. This study used two sets of publicly available prostate cancer MRI\u2009=\u2009radiomics data (Dataset 1: <jats:italic>n<\/jats:italic>\u2009=\u2009260; Dataset 2: <jats:italic>n<\/jats:italic>\u2009=\u2009100) with Gleason score clinical references. Each dataset was divided into training and holdout testing datasets at a ratio of 7:3 and analysed independently. The clinical references of the training set were permuted at different levels (increments of 5%) and repeated 20 times. Four feature selection algorithms and two classifiers were used to construct the models. Cross-validation was employed for training, while a separate hold-out testing set was used for evaluation. The Jaccard similarity coefficient was used to evaluate feature selection, while the area under the curve (AUC) and accuracy were used to assess model performance. An analysis of variance test with Bonferroni correction was conducted to compare the metrics of each model. The consistency of the feature selection performance decreased substantially with the clinical reference permutation. AUCs of the trained models with permutation particularly after 20% were significantly lower (Dataset 1 (with\u2009\u2265\u200920% permutation): 0.67, and Dataset 2 (\u2265\u200920% permutation): 0.74), compared to the AUC of models without permutation (Dataset 1: 0.94, Dataset 2: 0.97). The performances of the models were also associated with larger uncertainties and an increasing number of permuted clinical references. Clinical reference uncertainty can substantially influence MRI radiomic feature selection and modelling. The high accuracy of clinical references should be helpful in building reliable and robust radiomic models. Careful interpretation of the model performance is necessary, particularly for high-dimensional data.\n<\/jats:p>","DOI":"10.1186\/s42492-024-00180-9","type":"journal-article","created":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T03:43:47Z","timestamp":1731987827000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Computational analysis of variability and uncertainty in the clinical reference on magnetic resonance imaging radiomics: modelling and performance"],"prefix":"10.1186","volume":"7","author":[{"given":"Cindy","family":"Xue","sequence":"first","affiliation":[]},{"given":"Jing","family":"Yuan","sequence":"additional","affiliation":[]},{"given":"Gladys G.","family":"Lo","sequence":"additional","affiliation":[]},{"given":"Darren M.C.","family":"Poon","sequence":"additional","affiliation":[]},{"given":"Winnie CW","family":"Chu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"issue":"4","key":"180_CR1","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1016\/j.ejca.2011.11.036","volume":"48","author":"P Lambin","year":"2012","unstructured":"Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RGPM, Granton P et al (2012) Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 48(4):441\u2013446. https:\/\/doi.org\/10.1016\/j.ejca.2011.11.036","journal-title":"Eur J Cancer"},{"issue":"2","key":"180_CR2","doi-asserted-by":"publisher","first-page":"563","DOI":"10.1148\/radiol.2015151169","volume":"278","author":"RJ Gillies","year":"2016","unstructured":"Gillies RJ, Kinahan PE, Hricak H (2016) Radiomics: images are more than pictures, they are data. Radiology 278(2):563\u2013577. https:\/\/doi.org\/10.1148\/radiol.2015151169","journal-title":"Radiology"},{"issue":"5","key":"180_CR3","doi-asserted-by":"publisher","first-page":"e185","DOI":"10.1002\/mp.13678","volume":"47","author":"M Avanzo","year":"2020","unstructured":"Avanzo M, Wei LS, Stancanello J, Valli\u00e8res M, Rao A, Morin O et al (2020) Machine and deep learning methods for radiomics. Med Phys 47(5):e185\u2013e202. https:\/\/doi.org\/10.1002\/mp.13678","journal-title":"Med Phys"},{"issue":"1","key":"180_CR4","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1186\/s13244-022-01170-2","volume":"13","author":"A Demircio\u011flu","year":"2022","unstructured":"Demircio\u011flu A (2022) Evaluation of the dependence of radiomic features on the machine learning model. Insights Imaging 13(1):28. https:\/\/doi.org\/10.1186\/s13244-022-01170-2","journal-title":"Insights Imaging"},{"key":"180_CR5","doi-asserted-by":"publisher","DOI":"10.1038\/srep13087","volume":"5","author":"C Parmar","year":"2015","unstructured":"Parmar C, Grossmann P, Bussink J, Lambin P, Aerts HJWL (2015) Machine learning methods for quantitative radiomic biomarkers. Sci Rep 5:13087. https:\/\/doi.org\/10.1038\/srep13087","journal-title":"Sci Rep"},{"issue":"1","key":"180_CR6","doi-asserted-by":"publisher","first-page":"2085","DOI":"10.1038\/s41598-021-81272-x","volume":"11","author":"GA Nketiah","year":"2021","unstructured":"Nketiah GA, Elschot M, Scheenen TW, Maas MC, Bathen TF, Seln\u00e6s KM et al (2021) Utility of T2-weighted MRI texture analysis in assessment of peripheral zone prostate cancer aggressiveness: a single-arm, multicenter study. Sci Rep 11(1):2085","journal-title":"Sci Rep"},{"key":"180_CR7","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.ejrad.2019.06.019","volume":"117","author":"Y Yuan","year":"2019","unstructured":"Yuan Y, Ren JL, Shi YQ, Tao XF (2019) MRI-based radiomic signature as predictive marker for patients with head and neck squamous cell carcinoma. Eur J Radiol 117:193\u2013198. https:\/\/doi.org\/10.1016\/j.ejrad.2019.06.019","journal-title":"Eur J Radiol"},{"issue":"6","key":"180_CR8","doi-asserted-by":"publisher","first-page":"1518","DOI":"10.1002\/jmri.26010","volume":"48","author":"Q Tian","year":"2018","unstructured":"Tian Q, Yan LF, Zhang X, Zhang X, Hu YC, Han Y et al (2018) Radiomics strategy for glioma grading using texture features from multiparametric MRI. J Magn Reson Imaging 48(6):1518\u20131528","journal-title":"J Magn Reson Imaging"},{"issue":"10","key":"180_CR9","doi-asserted-by":"publisher","first-page":"4431","DOI":"10.21037\/qims-21-86","volume":"11","author":"C Xue","year":"2021","unstructured":"Xue C, Yuan J, Lo GG, Chang ATY, Poon DMC, Wong OL et al (2021) Radiomics feature reliability assessed by intraclass correlation coefficient: a systematic review. Quant Imaging Med Surg 11(10):4431\u20134460. https:\/\/doi.org\/10.21037\/qims-21-86","journal-title":"Quant Imaging Med Surg"},{"issue":"4","key":"180_CR10","doi-asserted-by":"publisher","first-page":"1117","DOI":"10.1016\/j.ijrobp.2018.05.022","volume":"102","author":"S Reuz\u00e9","year":"2018","unstructured":"Reuz\u00e9 S, Schernberg A, Orlhac F, Sun R, Chargari C, Dercle L et al (2018) Radiomics in nuclear medicine applied to radiation therapy: methods, pitfalls, and challenges. Int J Radiat Oncol Biol Phys 102(4):1117\u20131142","journal-title":"Int J Radiat Oncol Biol Phys"},{"issue":"1","key":"180_CR11","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/s42492-022-00106-3","volume":"5","author":"C Xue","year":"2022","unstructured":"Xue C, Yuan J, Zhou YH, Wong OL, Cheung KY, Yu SK (2022) Acquisition repeatability of MRI radiomics features in the head and neck: a dual-3D-sequence multi-scan study. Vis Comput Ind Biomed Art 5(1):10. https:\/\/doi.org\/10.1186\/s42492-022-00106-3","journal-title":"Vis Comput Ind Biomed Art"},{"key":"180_CR12","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/j.radonc.2019.03.001","volume":"135","author":"S Fiset","year":"2019","unstructured":"Fiset S, Welch ML, Weiss J, Pintilie M, Conway JL, Milosevic M et al (2019) Repeatability and reproducibility of MRI-based radiomic features in cervical cancer. Radiother Oncol 135:107\u2013114","journal-title":"Radiother Oncol"},{"key":"180_CR13","doi-asserted-by":"publisher","unstructured":"Yuan J, Xue C, Lo G, Wong OL, Zhou YH, Yu SK et al (2021) Quantitative assessment of acquisition imaging parameters on MRI radiomics features: a prospective anthropomorphic phantom study using a 3D-T2W-TSE sequence for MR-guided-radiotherapy. Quant Imaging Med Surg 11(5):1870\u20131887. https:\/\/doi.org\/10.21037\/qims-20-865","DOI":"10.21037\/qims-20-865"},{"key":"180_CR14","doi-asserted-by":"publisher","unstructured":"Bologna M, Corino V, Tenconi C, Facchinetti N, Calareso G, Iacovelli N et al (2020) Methodology and technology for the development of a prognostic MRI-based radiomic model for the outcome of head and neck cancer patients. In: Proceedings of the 42nd annual international conference of the IEEE engineering in medicine & biology society, IEEE, Montreal, 20-24 July 2020. https:\/\/doi.org\/10.1109\/EMBC44109.2020.9176565","DOI":"10.1109\/EMBC44109.2020.9176565"},{"issue":"3","key":"180_CR15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0213459","volume":"14","author":"L Duron","year":"2019","unstructured":"Duron L, Balvay D, Vande Perre S, Bouchouicha A, Savatovsky J, Sadik JC et al (2019) Gray-level discretization impacts reproducible MRI radiomics texture features. PLoS One 14(3):e0213459. https:\/\/doi.org\/10.1371\/journal.pone.0213459","journal-title":"PLoS One"},{"issue":"8","key":"180_CR16","doi-asserted-by":"publisher","first-page":"687","DOI":"10.5858\/1999-123-0687-DUEIPN","volume":"123","author":"DA Novis","year":"1999","unstructured":"Novis DA, Zarbo RJ, Valenstein PA (1999). Diagnostic uncertainty expressed in prostate needle biopsies: a College of American Pathologists Q-probes Study of 15 753 prostate needle biopsies in 332 institutions. Arch Pathol Lab Med 123(8):687\u2013692. https:\/\/doi.org\/10.5858\/1999-123-0687-DUEIPN","journal-title":"Arch Pathol Lab Med"},{"issue":"7","key":"180_CR17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0217702","volume":"14","author":"J Toivonen","year":"2019","unstructured":"Toivonen J, Montoya Perez I, Movahedi P, Merisaari H, Pesola M, Taimen P et al (2019) Radiomics and machine learning of multisequence multiparametric prostate MRI: towards improved non-invasive prostate cancer characterization. PLoS One 14(7):e0217702. https:\/\/doi.org\/10.1371\/journal.pone.0217702","journal-title":"PLoS One"},{"issue":"8","key":"180_CR18","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0237587","volume":"15","author":"Y Song","year":"2020","unstructured":"Song Y, Zhang J, Zhang YD, Hou Y, Yan X, Wang YD et al (2020) FeAture Explorer (FAE): a tool for developing and comparing radiomics models. PLoS One 15(8):e0237587. https:\/\/doi.org\/10.1371\/journal.pone.0237587","journal-title":"PLoS One"},{"issue":"1","key":"180_CR19","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1111\/j.1365-2559.2011.04003.x","volume":"60","author":"B Delahunt","year":"2012","unstructured":"Delahunt B, Miller RJ, Srigley JR, Evans AJ, Samaratunga H (2012) Gleason grading: past, present and future. Histopathology 60(1):75\u201386. https:\/\/doi.org\/10.1111\/j.1365-2559.2011.04003.x","journal-title":"Histopathology"},{"issue":"2","key":"180_CR20","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1111\/j.1469-8137.1912.tb05611.x","volume":"11","author":"P Jaccard","year":"1912","unstructured":"Jaccard P (1912) The distribution of the flora in the alpine zone. New Phytol 11(2):37\u201350. https:\/\/doi.org\/10.1111\/j.1469-8137.1912.tb05611.x","journal-title":"New Phytol"},{"issue":"2","key":"180_CR21","first-page":"627","volume":"4","author":"K Hajian-Tilaki","year":"2013","unstructured":"Hajian-Tilaki K (2013) Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med 4(2):627\u2013635","journal-title":"Caspian J Intern Med"},{"issue":"20","key":"180_CR22","doi-asserted-by":"publisher","first-page":"1920","DOI":"10.1161\/CIRCULATIONAHA.115.001593","volume":"132","author":"RC Deo","year":"2015","unstructured":"Deo RC (2015) Machine learning in medicine. Circulation 132(20):1920\u20131930. https:\/\/doi.org\/10.1161\/CIRCULATIONAHA.115.001593","journal-title":"Circulation"},{"issue":"23","key":"180_CR23","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics13233580","volume":"13","author":"C Xue","year":"2023","unstructured":"Xue C, Yuan J, Lo GG, Poon DMC, Chu WCW (2023) Evaluation of the reliability and the performance of magnetic resonance imaging radiomics in the presence of randomly generated irrelevant features for prostate cancer. Diagnostics 13(23):3580. https:\/\/doi.org\/10.3390\/diagnostics13233580","journal-title":"Diagnostics"},{"issue":"6245","key":"180_CR24","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1126\/science.aaa8415","volume":"349","author":"MI Jordan","year":"2015","unstructured":"Jordan MI, Mitchell TM (2015) Machine learning: trends, perspectives, and prospects. Science 349(6245):255\u2013260. https:\/\/doi.org\/10.1126\/science.aaa8415","journal-title":"Science"},{"key":"180_CR25","first-page":"1157","volume":"3","author":"I Guyon","year":"2003","unstructured":"Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157\u20131182","journal-title":"J Mach Learn Res"},{"issue":"5","key":"180_CR26","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1016\/j.compbiomed.2010.03.005","volume":"40","author":"DC Li","year":"2010","unstructured":"Li DC, Liu CW, Hu SC (2010) A learning method for the class imbalance problem with medical data sets. Comput Biol Med 40(5):509\u2013518. https:\/\/doi.org\/10.1016\/j.compbiomed.2010.03.005","journal-title":"Comput Biol Med"}],"container-title":["Visual Computing for Industry, Biomedicine, and Art"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s42492-024-00180-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s42492-024-00180-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s42492-024-00180-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T05:04:13Z","timestamp":1731992653000},"score":1,"resource":{"primary":{"URL":"https:\/\/vciba.springeropen.com\/articles\/10.1186\/s42492-024-00180-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["180"],"URL":"https:\/\/doi.org\/10.1186\/s42492-024-00180-9","relation":{},"ISSN":["2524-4442"],"issn-type":[{"type":"electronic","value":"2524-4442"}],"subject":[],"published":{"date-parts":[[2024,11,19]]},"assertion":[{"value":"1 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 October 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 November 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"28"}}