{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T22:59:33Z","timestamp":1777503573693,"version":"3.51.4"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,4,17]],"date-time":"2023-04-17T00:00:00Z","timestamp":1681689600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,17]],"date-time":"2023-04-17T00:00:00Z","timestamp":1681689600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"crossref","award":["952159"],"award-info":[{"award-number":["952159"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>There is a growing piece of evidence that artificial intelligence may be helpful in the entire prostate cancer disease continuum. However, building machine learning algorithms robust to inter- and intra-radiologist segmentation variability is still a challenge. With this goal in mind, several model training approaches were compared: removing unstable features according to the intraclass correlation coefficient (ICC); training independently with features extracted from each radiologist\u2019s mask; training with the feature average between both radiologists; extracting radiomic features from the intersection or union of masks; and creating a heterogeneous dataset by randomly selecting one of the radiologists\u2019 masks for each patient. The classifier trained with this last resampled dataset presented with the lowest generalization error, suggesting that training with heterogeneous data leads to the development of the most robust classifiers. On the contrary, removing features with low ICC resulted in the highest generalization error. The selected radiomics dataset, with the randomly chosen radiologists, was concatenated with deep features extracted from neural networks trained to segment the whole prostate. This new hybrid dataset was then used to train a classifier. The results revealed that, even though the hybrid classifier was less overfitted than the one trained with deep features, it still was unable to outperform the radiomics model.<\/jats:p>","DOI":"10.1038\/s41598-023-33339-0","type":"journal-article","created":{"date-parts":[[2023,4,17]],"date-time":"2023-04-17T10:12:23Z","timestamp":1681726343000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["Value of handcrafted and deep radiomic features towards training robust machine learning classifiers for prediction of prostate cancer disease aggressiveness"],"prefix":"10.1038","volume":"13","author":[{"given":"Ana","family":"Rodrigues","sequence":"first","affiliation":[]},{"given":"Nuno","family":"Rodrigues","sequence":"additional","affiliation":[]},{"given":"Jo\u00e3o","family":"Santinha","sequence":"additional","affiliation":[]},{"given":"Maria V.","family":"Lisitskaya","sequence":"additional","affiliation":[]},{"given":"Aycan","family":"Uysal","sequence":"additional","affiliation":[]},{"given":"Celso","family":"Matos","sequence":"additional","affiliation":[]},{"given":"In\u00eas","family":"Domingues","sequence":"additional","affiliation":[]},{"given":"Nickolas","family":"Papanikolaou","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,4,17]]},"reference":[{"key":"33339_CR1","doi-asserted-by":"publisher","first-page":"209","DOI":"10.3322\/caac.21660","volume":"71","author":"H Sung","year":"2021","unstructured":"Sung, H. et al. Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 71, 209\u2013249 (2021).","journal-title":"CA Cancer J. Clin."},{"key":"33339_CR2","doi-asserted-by":"publisher","first-page":"947","DOI":"10.1016\/j.juro.2017.10.021","volume":"199","author":"JA Halpern","year":"2018","unstructured":"Halpern, J. A. et al. Use of digital rectal examination as an adjunct to prostate specific antigen in the detection of clinically significant prostate cancer. J. Urol. 199, 947\u2013953 (2018).","journal-title":"J. Urol."},{"key":"33339_CR3","doi-asserted-by":"publisher","first-page":"175628722211090","DOI":"10.1177\/17562872221109020","volume":"14","author":"M Ferro","year":"2022","unstructured":"Ferro, M. et al. Radiomics in prostate cancer: An up-to-date review. Ther. Adv. Urol. 14, 17562872221109020 (2022).","journal-title":"Ther. Adv. Urol."},{"key":"33339_CR4","doi-asserted-by":"publisher","first-page":"1296","DOI":"10.1007\/s11547-021-01389-x","volume":"126","author":"C Scapicchio","year":"2021","unstructured":"Scapicchio, C. et al. A deep look into radiomics. Radiol. Med. (Torino) 126, 1296\u20131311 (2021).","journal-title":"Radiol. Med. (Torino)"},{"key":"33339_CR5","doi-asserted-by":"publisher","first-page":"959","DOI":"10.3390\/diagnostics11060959","volume":"11","author":"JJ Twilt","year":"2021","unstructured":"Twilt, J. J., van Leeuwen, K. G., Huisman, H. J., F\u00fctterer, J. J. & de Rooij, M. Artificial intelligence based algorithms for prostate cancer classification and detection on magnetic resonance imaging: A narrative review. Diagnostics 11, 959 (2021).","journal-title":"Diagnostics"},{"key":"33339_CR6","doi-asserted-by":"publisher","first-page":"34","DOI":"10.3390\/jimaging7020034","volume":"7","author":"G Cutaia","year":"2021","unstructured":"Cutaia, G. et al. Radiomics and prostate mri: Current role and future applications. J. Imaging 7, 34 (2021).","journal-title":"J. Imaging"},{"key":"33339_CR7","doi-asserted-by":"publisher","first-page":"1829","DOI":"10.3390\/diagnostics11101829","volume":"11","author":"F Midiri","year":"2021","unstructured":"Midiri, F., Vernuccio, F., Purpura, P., Alongi, P. & Bartolotta, T. V. Multiparametric mri and radiomics in prostate cancer: A review of the current literature. Diagnostics 11, 1829 (2021).","journal-title":"Diagnostics"},{"key":"33339_CR8","doi-asserted-by":"publisher","first-page":"716","DOI":"10.1007\/s00330-020-07105-z","volume":"31","author":"SG Gugliandolo","year":"2021","unstructured":"Gugliandolo, S. G. et al. Mri-based radiomics signature for localized prostate cancer: A new clinical tool for cancer aggressiveness prediction? Sub-study of prospective phase ii trial on ultra-hypofractionated radiotherapy (airc ig-13218). Eur. Radiol. 31, 716\u2013728 (2021).","journal-title":"Eur. Radiol."},{"key":"33339_CR9","volume":"5","author":"D Kwon","year":"2018","unstructured":"Kwon, D. et al. Classification of suspicious lesions on prostate multiparametric mri using machine learning. J. Med. Imaging 5, 034502 (2018).","journal-title":"J. Med. Imaging"},{"key":"33339_CR10","first-page":"25","volume":"11","author":"T Li","year":"2021","unstructured":"Li, T. et al. Development and validation of a radiomics nomogram for predicting clinically significant prostate cancer in pi-rads 3 lesions. Front. Oncol. 11, 25 (2021).","journal-title":"Front. Oncol."},{"key":"33339_CR11","doi-asserted-by":"publisher","first-page":"1102","DOI":"10.1002\/jmri.27132","volume":"52","author":"L Gong","year":"2020","unstructured":"Gong, L. et al. Noninvasive prediction of high-grade prostate cancer via biparametric mri radiomics. J. Magn. Reson. Imaging 52, 1102\u20131109 (2020).","journal-title":"J. Magn. Reson. Imaging"},{"key":"33339_CR12","doi-asserted-by":"publisher","first-page":"1767","DOI":"10.3390\/cancers12071767","volume":"12","author":"P Wo\u017anicki","year":"2020","unstructured":"Wo\u017anicki, P. et al. Multiparametric mri for prostate cancer characterization: Combined use of radiomics model with pi-rads and clinical parameters. Cancers 12, 1767 (2020).","journal-title":"Cancers"},{"key":"33339_CR13","doi-asserted-by":"publisher","first-page":"6757","DOI":"10.1007\/s00330-020-07064-5","volume":"30","author":"S Bernatz","year":"2020","unstructured":"Bernatz, S. et al. Comparison of machine learning algorithms to predict clinically significant prostate cancer of the peripheral zone with multiparametric mri using clinical assessment categories and radiomic features. Eur. Radiol. 30, 6757\u20136769 (2020).","journal-title":"Eur. Radiol."},{"key":"33339_CR14","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.ejrad.2017.11.001","volume":"98","author":"J Li","year":"2018","unstructured":"Li, J. et al. Support vector machines (svm) classification of prostate cancer gleason score in central gland using multiparametric magnetic resonance images: A cross-validated study. Eur. J. Radiol. 98, 61\u201367 (2018).","journal-title":"Eur. J. Radiol."},{"key":"33339_CR15","doi-asserted-by":"publisher","first-page":"S53","DOI":"10.1016\/S0167-8140(15)30242-5","volume":"111","author":"P Steenbergen","year":"2014","unstructured":"Steenbergen, P. et al. Prostate tumor delineation using multiparametric mri: Inter observer variability and pathology validation. Radiother. Oncol. 111, S53\u2013S54 (2014).","journal-title":"Radiother. Oncol."},{"key":"33339_CR16","first-page":"826","volume":"2021","author":"B Zhao","year":"2021","unstructured":"Zhao, B. Understanding sources of variation to improve the reproducibility of radiomics. Front. Oncol. 2021, 826 (2021).","journal-title":"Front. Oncol."},{"key":"33339_CR17","doi-asserted-by":"publisher","first-page":"289","DOI":"10.3390\/diagnostics12020289","volume":"12","author":"H Li","year":"2022","unstructured":"Li, H. et al. Machine learning in prostate mri for prostate cancer: Current status and future opportunities. Diagnostics 12, 289 (2022).","journal-title":"Diagnostics"},{"key":"33339_CR18","doi-asserted-by":"publisher","first-page":"12","DOI":"10.3390\/cancers14010012","volume":"14","author":"TJM Castillo","year":"2021","unstructured":"Castillo, T. J. M. et al. Classification of clinically significant prostate cancer on multi-parametric mri: A validation study comparing deep learning and radiomics. Cancers 14, 12 (2021).","journal-title":"Cancers"},{"key":"33339_CR19","doi-asserted-by":"publisher","first-page":"802964","DOI":"10.3389\/fonc.2021.802964","volume":"11","author":"E Bertelli","year":"2021","unstructured":"Bertelli, E. et al. Machine and deep learning prediction of prostate cancer aggressiveness using multiparametric mri. Front. Oncol. 11, 802964\u2013802964 (2021).","journal-title":"Front. Oncol."},{"key":"33339_CR20","doi-asserted-by":"publisher","unstructured":"Karssemeijer, G. L. O. D. J. B.\u00a0N. & Huisman., H. Prostatex challenge data, the cancer imaging archive. (2017). https:\/\/doi.org\/10.7937\/K9TCIA.2017.MURS5CL.","DOI":"10.7937\/K9TCIA.2017.MURS5CL"},{"key":"33339_CR21","doi-asserted-by":"publisher","first-page":"1083","DOI":"10.1109\/TMI.2014.2303821","volume":"33","author":"G Litjens","year":"2014","unstructured":"Litjens, G., Debats, O., Barentsz, J., Karssemeijer, N. & Huisman, H. Computer-aided detection of prostate cancer in mri. IEEE Trans. Med. Imaging 33, 1083\u20131092. https:\/\/doi.org\/10.1109\/TMI.2014.2303821 (2014).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"33339_CR22","doi-asserted-by":"publisher","first-page":"1045","DOI":"10.1007\/s10278-013-9622-7","volume":"26","author":"K Clark","year":"2013","unstructured":"Clark, K. et al. The cancer imaging archive (tcia): Maintaining and operating a public information repository. J. Digit. Imaging 26, 1045\u20131057 (2013).","journal-title":"J. Digit. Imaging"},{"key":"33339_CR23","doi-asserted-by":"publisher","first-page":"1310","DOI":"10.1109\/TMI.2010.2046908","volume":"29","author":"NJ Tustison","year":"2010","unstructured":"Tustison, N. J. et al. N4itk: Improved n3 bias correction. IEEE Trans. Med. Imaging 29, 1310\u20131320 (2010).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"33339_CR24","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1007\/s10278-017-0037-8","volume":"31","author":"Z Yaniv","year":"2018","unstructured":"Yaniv, Z., Lowekamp, B. C., Johnson, H. J. & Beare, R. Simpleitk image-analysis notebooks: A collaborative environment for education and reproducible research. J. Digit. Imaging 31, 290\u2013303 (2018).","journal-title":"J. Digit. Imaging"},{"key":"33339_CR25","doi-asserted-by":"publisher","first-page":"e104","DOI":"10.1158\/0008-5472.CAN-17-0339","volume":"77","author":"JJ Van Griethuysen","year":"2017","unstructured":"Van Griethuysen, J. J. et al. Computational radiomics system to decode the radiographic phenotype. Can. Res. 77, e104\u2013e107 (2017).","journal-title":"Can. Res."},{"key":"33339_CR26","unstructured":"Van Rossum, G. & Drake, F. L. Python 3 Reference Manual (CreateSpace, 2009)."},{"key":"33339_CR27","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","volume":"15","author":"TK Koo","year":"2016","unstructured":"Koo, T. K. & Li, M. Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155\u2013163 (2016).","journal-title":"J. Chiropr. Med."},{"key":"33339_CR28","doi-asserted-by":"publisher","first-page":"1467","DOI":"10.3390\/cancers15051467","volume":"15","author":"NM Rodrigues","year":"2023","unstructured":"Rodrigues, N. M., Silva, S., Vanneschi, L. & Papanikolaou, N. A comparative study of automated deep learning segmentation models for prostate mri. Cancers 15, 1467 (2023).","journal-title":"Cancers"},{"key":"33339_CR29","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. arXiv:abs\/1505.04597 (2015).","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"33339_CR30","doi-asserted-by":"crossref","unstructured":"Zhou, Z., Siddiquee, M. M.\u00a0R., Tajbakhsh, N. & Liang, J. Unet++: A Nested u-net Architecture for Medical Image Segmentation. (2018). arXiv:1807.10165.","DOI":"10.1007\/978-3-030-00889-5_1"},{"key":"33339_CR31","unstructured":"Oktay, O. et\u00a0al. Attention u-net: Learning Where to Look for the Pancreas. (2018). arXiv:abs\/1804.03999."},{"key":"33339_CR32","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1038\/s41598-020-71080-0","volume":"10","author":"N Aldoj","year":"2020","unstructured":"Aldoj, N., Biavati, F., Michallek, F., Stober, S. & Dewey, M. Automatic prostate and prostate zones segmentation of magnetic resonance images using densenet-like u-net. Sci. Rep. 10, 25 (2020).","journal-title":"Sci. Rep."},{"key":"33339_CR33","doi-asserted-by":"crossref","unstructured":"Alom, M.\u00a0Z., Hasan, M., Yakopcic, C., Taha, T.\u00a0M. & Asari, V.\u00a0K. Recurrent Residual Convolutional Neural Network Based on u-net (r2u-net) for Medical Image Segmentation. (2018). arXiv:1802.06955.","DOI":"10.1109\/NAECON.2018.8556686"},{"key":"33339_CR34","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"33339_CR35","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","volume":"585","author":"CR Harris","year":"2020","unstructured":"Harris, C. R. et al. Array programming with NumPy. Nature 585, 357\u2013362. https:\/\/doi.org\/10.1038\/s41586-020-2649-2 (2020).","journal-title":"Nature"},{"key":"33339_CR36","doi-asserted-by":"crossref","unstructured":"McKinney, W. et\u00a0al. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, vol. 445, 51\u201356 (Austin, TX, 2010).","DOI":"10.25080\/Majora-92bf1922-00a"},{"key":"33339_CR37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v028.i05","volume":"28","author":"M Kuhn","year":"2020","unstructured":"Kuhn, M. Building predictive models in r using the caret package. J. Stat. Softw. 28, 1\u201326. https:\/\/doi.org\/10.18637\/jss.v028.i05 (2020).","journal-title":"J. Stat. Softw."},{"key":"33339_CR38","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. Smote: Synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321\u2013357 (2002).","journal-title":"J. Artif. Intell. Res."},{"key":"33339_CR39","doi-asserted-by":"crossref","unstructured":"Breiman, L. Random forests. Mach. Learn. 45, 5\u201332 (2001).","DOI":"10.1023\/A:1010933404324"},{"key":"33339_CR40","doi-asserted-by":"publisher","first-page":"56","DOI":"10.21105\/joss.00638","volume":"3","author":"S Raschka","year":"2018","unstructured":"Raschka, S. Mlxtend: Providing machine learning and data science utilities and extensions to python\u2019s scientific computing stack. J. Open Sourc. Softw. 3, 56. https:\/\/doi.org\/10.21105\/joss.00638 (2018).","journal-title":"J. Open Sourc. Softw."},{"key":"33339_CR41","first-page":"25","volume":"30","author":"SM Lundberg","year":"2017","unstructured":"Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30, 25 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"33339_CR42","doi-asserted-by":"publisher","first-page":"1892","DOI":"10.1002\/jmri.27565","volume":"54","author":"C Han","year":"2021","unstructured":"Han, C. et al. Radiomics models based on apparent diffusion coefficient maps for the prediction of high-grade prostate cancer at radical prostatectomy: Comparison with preoperative biopsy. J. Magn. Reson. Imaging 54, 1892\u20131901 (2021).","journal-title":"J. Magn. Reson. Imaging"},{"key":"33339_CR43","doi-asserted-by":"publisher","first-page":"6065","DOI":"10.3390\/cancers13236065","volume":"13","author":"A Rodrigues","year":"2021","unstructured":"Rodrigues, A. et al. Prediction of prostate cancer disease aggressiveness using bi-parametric mri radiomics. Cancers 13, 6065 (2021).","journal-title":"Cancers"},{"key":"33339_CR44","doi-asserted-by":"publisher","first-page":"549","DOI":"10.2214\/AJR.17.18494","volume":"210","author":"X-K Niu","year":"2018","unstructured":"Niu, X.-K. et al. Clinical application of biparametric mri texture analysis for detection and evaluation of high-grade prostate cancer in zone-specific regions. Am. J. Roentgenol. 210, 549\u2013556 (2018).","journal-title":"Am. J. Roentgenol."},{"key":"33339_CR45","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1016\/j.ejrad.2019.02.032","volume":"114","author":"M Xu","year":"2019","unstructured":"Xu, M. et al. Using biparametric mri radiomics signature to differentiate between benign and malignant prostate lesions. Eur. J. Radiol. 114, 38\u201344 (2019).","journal-title":"Eur. J. Radiol."},{"key":"33339_CR46","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1038\/nrclinonc.2017.141","volume":"14","author":"P Lambin","year":"2017","unstructured":"Lambin, P. et al. Radiomics: The bridge between medical imaging and personalized medicine. Nat. Rev. Clin. Oncol. 14, 749\u2013762 (2017).","journal-title":"Nat. Rev. Clin. Oncol."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-33339-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-33339-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-33339-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,17]],"date-time":"2023-04-17T10:14:45Z","timestamp":1681726485000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-33339-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,17]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["33339"],"URL":"https:\/\/doi.org\/10.1038\/s41598-023-33339-0","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,17]]},"assertion":[{"value":"3 January 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 April 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 April 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"6206"}}