{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T01:37:50Z","timestamp":1775871470830,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T00:00:00Z","timestamp":1691452800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Vice Chancellery for Research and Technology, Isfahan University of Medical Sciences","award":["198337"],"award-info":[{"award-number":["198337"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Accurate detection of respiratory system damage including COVID-19 is considered one of the crucial applications of deep learning (DL) models using CT images. However, the main shortcoming of the published works has been unreliable reported accuracy and the lack of repeatability with new datasets, mainly due to slice-wise splits of the data, creating dependency between training and test sets due to shared data across the sets. We introduce a new dataset of CT images (ISFCT Dataset) with labels indicating the subject-wise split to train and test our DL algorithms in an unbiased manner. We also use this dataset to validate the real performance of the published works in a subject-wise data split. Another key feature provides more specific labels (eight characteristic lung features) rather than being limited to COVID-19 and healthy labels. We show that the reported high accuracy of the existing models on current slice-wise splits is not repeatable for subject-wise splits, and distribution differences between data splits are demonstrated using t-distribution stochastic neighbor embedding. We indicate that, by examining subject-wise data splitting, less complicated models show competitive results compared to the exiting complicated models, demonstrating that complex models do not necessarily generate accurate and repeatable results.<\/jats:p>","DOI":"10.3390\/jimaging9080159","type":"journal-article","created":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T12:35:42Z","timestamp":1691498142000},"page":"159","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Biased Deep Learning Methods in Detection of COVID-19 Using CT Images: A Challenge Mounted by Subject-Wise-Split ISFCT Dataset"],"prefix":"10.3390","volume":"9","author":[{"given":"Shiva","family":"Parsarad","sequence":"first","affiliation":[{"name":"Medical Image and Signal Processing Research Center, School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"},{"name":"Law, Economics, and Data Science Group, Department of Humanities, Social and Political Science, ETH Zurich, 8092 Zurich, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Narges","family":"Saeedizadeh","sequence":"additional","affiliation":[{"name":"Medical Image and Signal Processing Research Center, School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"},{"name":"Institute for Intelligent Systems Research and Innovation, Deakin University, Melbourne, VIC 3125, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ghazaleh Jamalipour","family":"Soufi","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shamim","family":"Shafieyoon","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Farzaneh","family":"Hekmatnia","sequence":"additional","affiliation":[{"name":"St. George\u2019s Hospital, London SW17 0RE, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew Parviz","family":"Zarei","sequence":"additional","affiliation":[{"name":"St. George\u2019s Hospital, London SW17 0RE, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samira","family":"Soleimany","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amir","family":"Yousefi","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hengameh","family":"Nazari","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pegah","family":"Torabi","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1173-9989","authenticated-orcid":false,"given":"Abbas","family":"S. Milani","sequence":"additional","affiliation":[{"name":"School of Engineering, University of British Columbia, Kelowna, BC V1V 1V7, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seyed Ali","family":"Madani Tonekaboni","sequence":"additional","affiliation":[{"name":"Cyclica Inc., Toronto, ON M5J 1A7, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hossein","family":"Rabbani","sequence":"additional","affiliation":[{"name":"Medical Image and Signal Processing Research Center, School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ali","family":"Hekmatnia","sequence":"additional","affiliation":[{"name":"Department of Radiology, School of Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0087-9476","authenticated-orcid":false,"given":"Rahele","family":"Kafieh","sequence":"additional","affiliation":[{"name":"Medical Image and Signal Processing Research Center, School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan JM76+5M3, Iran"},{"name":"Department of Engineering, Durham University, Durham DH1 3LE, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/RBME.2020.2987975","article-title":"Review of artificial intelligence techniques in imaging data acquisition, segmentation, and diagnosis for COVID-19","volume":"14","author":"Shi","year":"2020","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1148\/radiol.2020200370","article-title":"Time course of lung changes at chest CT during recovery from coronavirus disease 2019 (COVID-19)","volume":"295","author":"Pan","year":"2020","journal-title":"Radiology"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Loey, M., Manogaran, G., and Khalifa, N.E.M. (2020). A deep transfer learning model with classical data augmentation and CGAN to detect COVID-19 from chest CT radiography digital images. Neural Comput. Appl., 1\u201313.","DOI":"10.1007\/s00521-020-05437-x"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Devnath, L., Luo, S., Summons, P., and Wang, D. (2020, January 12\u201315). Performance comparison of deep learning models for black lung detection on chest X-ray radiographs. Proceedings of the 3rd International Conference on Software Engineering and Information Management, Sydney, Australia.","DOI":"10.1145\/3378936.3378968"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"115041","DOI":"10.1109\/ACCESS.2020.3003810","article-title":"Iteratively pruned deep learning ensembles for COVID-19 detection in chest X-rays","volume":"8","author":"Rajaraman","year":"2020","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"gix019","DOI":"10.1093\/gigascience\/gix019","article-title":"The need to approximate the use-case in clinical machine learning","volume":"6","author":"Saeb","year":"2017","journal-title":"Gigascience"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1038\/s41746-019-0178-x","article-title":"Detecting the impact of subject characteristics on machine learning-based diagnostic applications","volume":"2","author":"Pratap","year":"2019","journal-title":"NPJ Digit. Med."},{"key":"ref_8","unstructured":"Yang, X., He, X., Zhao, J., Zhang, Y., Zhang, S., and Xie, P. (2020). COVID-CT-dataset: A CT scan dataset about COVID-19. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e27468","DOI":"10.2196\/27468","article-title":"Deep convolutional neural network\u2013based computer-aided detection system for COVID-19 using multiple lung scans: Design and implementation study","volume":"23","author":"Ghaderzadeh","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"14353","DOI":"10.1038\/s41598-021-93832-2","article-title":"Deep learning for COVID-19 detection based on CT images","volume":"11","author":"Zhao","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1038\/s41564-020-0695-z","article-title":"The species Severe acute respiratory syndrome-related coronavirus: Classifying 2019-nCoV and naming it SARS-CoV-2","volume":"5","author":"Gorbalenya","year":"2020","journal-title":"Nat. Microbiol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. (2020, January 23\u201328). Big Transfer (BiT): General Visual Representation Learning. Proceedings of the Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK.","DOI":"10.1007\/978-3-030-58558-7_29"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1423","DOI":"10.1016\/j.cell.2020.04.045","article-title":"Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography","volume":"181","author":"Zhang","year":"2020","journal-title":"Cell"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1016\/j.bbe.2021.05.013","article-title":"Automatic detection of coronavirus disease (COVID-19) in X-ray and CT images: A machine learning based approach","volume":"41","author":"Kassania","year":"2021","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Cohen, J.P., Morrison, P., Dao, L., Roth, K., Duong, T.Q., and Ghassemi, M. (2020). COVID-19 image data collection: Prospective predictions are the future. arXiv.","DOI":"10.59275\/j.melba.2020-48g7"},{"key":"ref_16","unstructured":"Kaggle, R. (2022, June 18). Pneumonia Detection Challenge. Available online: https:\/\/www.kaggle.com\/c\/rsna-pneumonia-detection-challenge."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Jaiswal, A.K., Tiwari, P., Rathi, V.K., Qian, J., Pandey, H.M., and Albuquerque, V.H.C. (2020). Covidpen: A novel COVID-19 detection model using chest x-rays and ct scans. Medrxiv.","DOI":"10.1101\/2020.07.08.20149161"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"7672196","DOI":"10.1155\/2022\/7672196","article-title":"COVID-19 detection based on lung ct scan using deep learning techniques","volume":"2022","author":"Kogilavani","year":"2022","journal-title":"Comput. Math. Methods Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s10522-021-09946-7","article-title":"COVID-19 detection in CT and CXR images using deep learning models","volume":"23","author":"Chouat","year":"2022","journal-title":"Biogerontology"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"132665","DOI":"10.1109\/ACCESS.2020.3010287","article-title":"Can AI help in screening viral and COVID-19 pneumonia?","volume":"8","author":"Chowdhury","year":"2020","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"104319","DOI":"10.1016\/j.compbiomed.2021.104319","article-title":"Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images","volume":"132","author":"Rahman","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1007\/s10439-022-02958-5","article-title":"Detection of COVID-19 from CT and Chest X-ray Images Using Deep Learning Models","volume":"50","author":"Zouch","year":"2022","journal-title":"Ann. Biomed. Eng."},{"key":"ref_23","unstructured":"(2023, June 02). Available online: https:\/\/github.com\/UCSD-AI4H\/COVID-CT\/tree\/master\/Images-processed."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1716","DOI":"10.1038\/s41598-022-05532-0","article-title":"Effective deep learning approaches for predicting COVID-19 outcomes from chest computed tomography volumes","volume":"12","author":"Ortiz","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"104092","DOI":"10.1016\/j.compbiomed.2020.104092","article-title":"The importance of standardisation\u2013COVID-19 CT & Radiograph Image Data Stock for deep learning purpose","volume":"127","author":"Misztal","year":"2020","journal-title":"Comput. Biol. Med."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e044640","DOI":"10.1136\/bmjopen-2020-044640","article-title":"Demographic risk factors for COVID-19 infection, severity, ICU admission and death: A meta-analysis of 59 studies","volume":"11","author":"Pijls","year":"2021","journal-title":"BMJ Open"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"152","DOI":"10.3389\/fpubh.2020.00152","article-title":"Gender differences in patients with COVID-19: Focus on severity and mortality","volume":"8","author":"Jin","year":"2020","journal-title":"Front. Public Health"},{"key":"ref_28","unstructured":"Gunderman, R.B. (2006). Essential Radiology: Clinical Presentation, Pathophysiology, Imaging, Thieme. [2nd ed.]."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1148\/rg.2018170048","article-title":"Radiographic and CT features of viral pneumonia","volume":"38","author":"Koo","year":"2018","journal-title":"Radiographics"},{"key":"ref_30","unstructured":"Hacking, C., Weerakkody, Y., and Bell, D. (2021). Left lower lobe consolidation. Radiopedia, 66157."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1509","DOI":"10.1148\/rg.236035101","article-title":"\u201cCrazy-paving\u201d pattern at thin-section CT of the lungs: Radiologic-pathologic overview","volume":"23","author":"Rossi","year":"2003","journal-title":"Radiographics"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1186\/s13244-019-0789-4","article-title":"Chest imaging using signs, symbols, and naturalistic images: A practical guide for radiologists and non-radiologists","volume":"10","author":"Chiarenza","year":"2019","journal-title":"Insights Imaging"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"100427","DOI":"10.1016\/j.imu.2020.100427","article-title":"COVID-19 detection in CT images with deep learning: A voting-based scheme and cross-datasets analysis","volume":"20","author":"Silva","year":"2020","journal-title":"Inform. Med. Unlocked"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_35","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_36","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 6\u201311). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, Lille, France."},{"key":"ref_37","unstructured":"Loshchilov, I., and Hutter, F. (2016, January 2\u20134). Stochastic gradient descent with warm restarts. Proceedings of the 5th International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_39","unstructured":"Zeiler, M.D., and Fergus, R. (2014). Computer Vision\u2013ECCV 2014: 13th European Conference on Computer Vision, Zurich, Switzerland, 6\u201312 September 2014, Springer."},{"key":"ref_40","unstructured":"Qiu Lin, Z., Javad Shafiee, M., Bochkarev, S., St Jules, M., Wang, X.Y., and Wong, A. (2019). Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.cobme.2018.12.005","article-title":"Cats or CAT scans: Transfer learning from natural or medical image source data sets?","volume":"9","author":"Cheplygina","year":"2019","journal-title":"Curr. Opin. Biomed. Eng."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/8\/159\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:28:02Z","timestamp":1760128082000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/9\/8\/159"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,8]]},"references-count":41,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["jimaging9080159"],"URL":"https:\/\/doi.org\/10.3390\/jimaging9080159","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,8]]}}}