{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T11:14:38Z","timestamp":1789038878801,"version":"build-2803163510"},"reference-count":29,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:00:00Z","timestamp":1687132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100004440","name":"DBT\/Wellcome Trust India Alliance Early Career Fellowship","doi-asserted-by":"publisher","award":["IA\/E\/18\/1\/504306"],"award-info":[{"award-number":["IA\/E\/18\/1\/504306"]}],"id":[{"id":"10.13039\/100004440","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004440","name":"DBT\/Wellcome Trust India Alliance Early Career Fellowship","doi-asserted-by":"publisher","award":["2022-23"],"award-info":[{"award-number":["2022-23"]}],"id":[{"id":"10.13039\/100004440","id-type":"DOI","asserted-by":"publisher"}]},{"name":"University of Surrey IAS fellowship for external academics","award":["IA\/E\/18\/1\/504306"],"award-info":[{"award-number":["IA\/E\/18\/1\/504306"]}]},{"name":"University of Surrey IAS fellowship for external academics","award":["2022-23"],"award-info":[{"award-number":["2022-23"]}]},{"name":"Alliance Medical Ltd.","award":["IA\/E\/18\/1\/504306"],"award-info":[{"award-number":["IA\/E\/18\/1\/504306"]}]},{"name":"Alliance Medical Ltd.","award":["2022-23"],"award-info":[{"award-number":["2022-23"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Applied Sciences"],"abstract":"<jats:p>Radiomics involves the extraction of information from medical images that are not visible to the human eye. There is evidence that these features can be used for treatment stratification and outcome prediction. However, there is much discussion about the reproducibility of results between different studies. This paper studies the reproducibility of CT texture features used in radiomics, comparing two feature extraction implementations, namely the MATLAB toolkit and Pyradiomics, when applied to independent datasets of CT scans of patients: (i) the open access RIDER dataset containing a set of repeat CT scans taken 15 min apart for 31 patients (RIDER Scan 1 and Scan 2, respectively) treated for lung cancer; and (ii) the open access HN1 dataset containing 137 patients treated for head and neck cancer. Gross tumor volume (GTV), manually outlined by an experienced observer available on both datasets, was used. The 43 common radiomics features available in MATLAB and Pyradiomics were calculated using two intensity-level quantization methods with and without an intensity threshold. Cases were ranked for each feature for all combinations of quantization parameters, and the Spearman\u2019s rank coefficient, rs, calculated. Reproducibility was defined when a highly correlated feature in the RIDER dataset also correlated highly in the HN1 dataset, and vice versa. A total of 29 out of the 43 reported stable features were found to be highly reproducible between MATLAB and Pyradiomics implementations, having a consistently high correlation in rank ordering for RIDER Scan 1 and RIDER Scan 2 (rs &gt; 0.8). 18\/43 reported features were common in the RIDER and HN1 datasets, suggesting they may be agnostic to disease site. Useful radiomics features should be selected based on reproducibility. This study identified a set of features that meet this requirement and validated the methodology for evaluating reproducibility between datasets.<\/jats:p>","DOI":"10.3390\/app13127291","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T02:26:56Z","timestamp":1687228016000},"page":"7291","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Reproducibility in Radiomics: A Comparison of Feature Extraction Methods and Two Independent Datasets"],"prefix":"10.3390","volume":"13","author":[{"given":"Hannah Mary T.","family":"Thomas","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Helen Y. C.","family":"Wang","sequence":"additional","affiliation":[{"name":"Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford GU2 7XH, UK"},{"name":"Department of Medical Physics, Royal Surrey NHS Foundation Trust, Guildford GU2 7XX, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2844-7611","authenticated-orcid":false,"given":"Amal Joseph","family":"Varghese","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ellen M.","family":"Donovan","sequence":"additional","affiliation":[{"name":"Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford GU2 7XH, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris P.","family":"South","sequence":"additional","affiliation":[{"name":"Department of Medical Physics, Royal Surrey NHS Foundation Trust, Guildford GU2 7XX, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Helen","family":"Saxby","sequence":"additional","affiliation":[{"name":"St Luke\u2019s Cancer Centre, Royal Surrey NHS Foundation Trust, Guildford GU2 7XX, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0652-808X","authenticated-orcid":false,"given":"Andrew","family":"Nisbet","sequence":"additional","affiliation":[{"name":"Department of Medical Physics and Biomedical Engineering, University College London, London WC1E 6BT, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vineet","family":"Prakash","sequence":"additional","affiliation":[{"name":"St Luke\u2019s Cancer Centre, Royal Surrey NHS Foundation Trust, Guildford GU2 7XX, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2814-161X","authenticated-orcid":false,"given":"Balu Krishna","family":"Sasidharan","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon Pradeep","family":"Pavamani","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Devakumar","family":"Devadhas","sequence":"additional","affiliation":[{"name":"Department of Nuclear Medicine, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manu","family":"Mathew","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajesh Gunasingam","family":"Isiah","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Christian Medical College Vellore, Vellore 632004, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8102-6581","authenticated-orcid":false,"given":"Philip M.","family":"Evans","sequence":"additional","affiliation":[{"name":"Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford GU2 7XH, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,19]]},"reference":[{"key":"ref_1","first-page":"1","article-title":"Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach","volume":"5","author":"Aerts","year":"2014","journal-title":"Nat. Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1016\/S1470-2045(17)30438-2","article-title":"The TNM classification of malignant tumours\u2014Towards common understanding and reasonable expectations","volume":"18","author":"Brierley","year":"2017","journal-title":"Lancet Oncol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/S0140-6736(09)60329-9","article-title":"Avoidable waste in the production and reporting of research evidence","volume":"374","author":"Chalmers","year":"2009","journal-title":"Lancet"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1143","DOI":"10.1016\/j.ijrobp.2018.05.053","article-title":"Repeatability and Reproducibility of Radiomic Features: A Systematic Review","volume":"102","author":"Traverso","year":"2018","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1593\/tlo.13844","article-title":"Reproducibility and Prognosis of Quantitative Features Extracted from CT Images","volume":"7","author":"Balagurunathan","year":"2014","journal-title":"Transl. Oncol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1016\/j.radonc.2016.04.004","article-title":"Radiomic phenotype features predict pathological response in non-small cell lung cancer","volume":"119","author":"Coroller","year":"2016","journal-title":"Radiother. Oncol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6784","DOI":"10.1118\/1.4934826","article-title":"Can radiomics features be reproducibly measured from CBCT images for patients with non-small cell lung cancer?","volume":"42","author":"Fave","year":"2015","journal-title":"Med. Phys."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"588","DOI":"10.1038\/s41598-017-00665-z","article-title":"Delta-radiomics features for the prediction of patient outcomes in non\u2013small cell lung cancer","volume":"7","author":"Fave","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"165012","DOI":"10.1088\/1361-6560\/ab23a7","article-title":"The stability of imaging biomarkers in radiomics: A framework for evaluation","volume":"64","author":"Wang","year":"2019","journal-title":"Phys. Med. Biol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"10117","DOI":"10.1038\/s41598-017-10371-5","article-title":"Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer","volume":"7","author":"Perrin","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"15TR03","DOI":"10.1088\/1361-6560\/ac7fd7","article-title":"Impact of image quality on radiomics applications","volume":"67","author":"Cui","year":"2022","journal-title":"Phys. Med. Biol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1148\/radiol.2522081593","article-title":"Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer","volume":"252","author":"Zhao","year":"2009","journal-title":"Radiology"},{"key":"ref_13","unstructured":"(2023, January 27). Welcome to Pyradiomics Documentation!\u2014Pyradiomics v3.0.1.post15+g2791e23 Documentation n.d. Available online: https:\/\/pyradiomics.readthedocs.io\/en\/latest\/."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2055","DOI":"10.1038\/s41598-021-81526-8","article-title":"Repeatability and reproducibility study of radiomic features on a phantom and human cohort","volume":"11","author":"Jha","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"981","DOI":"10.3174\/ajnr.A5139","article-title":"Quantitative Assessment of Variation in CT Parameters on Texture Features: Pilot Study Using a Nonanatomic Phantom","volume":"38","author":"Buch","year":"2017","journal-title":"Am. J. Neuroradiol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1148\/radiol.2018172361","article-title":"Radiomics of CT Features May Be Nonreproducible and Redundant: Influence of CT Acquisition Parameters","volume":"288","author":"Berenguer","year":"2018","journal-title":"Radiology"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4008","DOI":"10.1007\/s00330-018-5829-9","article-title":"CT reconstruction algorithms affect histogram and texture analysis: Evidence for liver parenchyma, focal solid liver lesions, and renal cysts","volume":"29","author":"Ahn","year":"2019","journal-title":"Eur. Radiol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"10545","DOI":"10.1038\/s41598-018-28895-9","article-title":"Voxel size and gray level normalization of CT radiomic features in lung cancer","volume":"8","author":"Latifi","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"100450","DOI":"10.1016\/j.phro.2023.100450","article-title":"Multi-centre radiomics for prediction of recurrence following radical radiotherapy for head and neck cancers: Consequences of feature selection, machine learning classifiers and batch-effect harmonization","volume":"26","author":"Varghese","year":"2023","journal-title":"Phys. Imaging Radiat. Oncol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"11542","DOI":"10.1038\/s41598-021-90985-y","article-title":"Reproducibility of CT-based radiomic features against image resampling and perturbations for tumour and healthy kidney in renal cancer patients","volume":"11","author":"Mottola","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_21","first-page":"826","article-title":"Understanding Sources of Variation to Improve the Reproducibility of Radiomics","volume":"11","author":"Zhao","year":"2021","journal-title":"Front. Oncol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/s13244-020-00887-2","article-title":"Radiomics in medical imaging\u2014\u201chow-to\u201d guide and critical reflection","volume":"11","author":"Cester","year":"2020","journal-title":"Insights Into Imaging"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3591","DOI":"10.1158\/1078-0432.CCR-12-1307","article-title":"Tumor heterogeneity and permeability as measured on the CT component of PET\/CT predict survival in patients with non-small cell lung cancer","volume":"19","author":"Win","year":"2013","journal-title":"Clin. Cancer Res."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.jtho.2016.11.2226","article-title":"Radiomic-Based Pathological Response Prediction from Primary Tumors and Lymph Nodes in NSCLC","volume":"12","author":"Coroller","year":"2017","journal-title":"J. Thorac. Oncol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1186\/s40537-023-00703-w","article-title":"Data-centric artificial intelligence in oncology: A systematic review assessing data quality in machine learning models for head and neck cancer","volume":"10","author":"Adeoye","year":"2023","journal-title":"J. Big Data"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2757","DOI":"10.1007\/s11517-022-02631-y","article-title":"A comprehensive survey on deep learning techniques in CT image quality improvement","volume":"60","author":"Li","year":"2022","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.radonc.2022.02.034","article-title":"Deep learning improves image quality and radiomics reproducibility for high-speed four-dimensional computed tomography reconstruction","volume":"170","author":"Yang","year":"2022","journal-title":"Radiother. Oncol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.jacr.2020.09.050","article-title":"In the Era of Deep Learning, Why Reconstruct an Image at All?","volume":"18","author":"Chung","year":"2021","journal-title":"J. Am. Coll. Radiol."},{"key":"ref_29","first-page":"287","article-title":"IBSI: An international community radiomics standardization initiative","volume":"59","author":"Hatt","year":"2018","journal-title":"J. Nucl. Med."}],"container-title":["Applied Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3417\/13\/12\/7291\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:56:31Z","timestamp":1760126191000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3417\/13\/12\/7291"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,19]]},"references-count":29,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["app13127291"],"URL":"https:\/\/doi.org\/10.3390\/app13127291","relation":{},"ISSN":["2076-3417"],"issn-type":[{"value":"2076-3417","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,19]]}}}