{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,30]],"date-time":"2025-12-30T03:44:12Z","timestamp":1767066252277,"version":"3.37.3"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,21]],"date-time":"2022-04-21T00:00:00Z","timestamp":1650499200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,21]],"date-time":"2022-04-21T00:00:00Z","timestamp":1650499200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["SFRH\/BD\/120435\/2016","SFRH\/BD\/146437\/2019","UIDB\/50014\/2020","071_596637133 CXR_AI4COVID-19"],"award-info":[{"award-number":["SFRH\/BD\/120435\/2016","SFRH\/BD\/146437\/2019","UIDB\/50014\/2020","071_596637133 CXR_AI4COVID-19"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The coronavirus disease 2019 (COVID-19) pandemic has impacted healthcare systems across the world. Chest radiography (CXR) can be used as a complementary method for diagnosing\/following COVID-19 patients. However, experience level and workload of technicians and radiologists may affect the decision process. Recent studies suggest that deep learning can be used to assess CXRs, providing an important second opinion for radiologists and technicians in the decision process, and super-human performance in detection of COVID-19 has been reported in multiple studies. In this study, the clinical applicability of deep learning systems for COVID-19 screening was assessed by testing the performance of deep learning systems for the detection of COVID-19. Specifically, four datasets were used: (1) a collection of multiple public datasets (284.793 CXRs); (2) BIMCV dataset (16.631 CXRs); (3) COVIDGR (852 CXRs) and 4) a private dataset (6.361 CXRs). All datasets were collected retrospectively and consist of only frontal CXR views. A ResNet-18 was trained on each of the datasets for the detection of COVID-19. It is shown that a high dataset bias was present, leading to high performance in intradataset train-test scenarios (area under the curve 0.55\u20130.84 on the collection of public datasets). Significantly lower performances were obtained in interdataset train-test scenarios however (area under the curve &gt; 0.98). A subset of the data was then assessed by radiologists for comparison to the automatic systems. Finetuning with radiologist annotations significantly increased performance across datasets (area under the curve 0.61\u20130.88) and improved the attention on clinical findings in positive COVID-19 CXRs. Nevertheless, tests on CXRs from different hospital services indicate that the screening performance of CXR and automatic systems is limited (area under the curve &lt; 0.6 on emergency service CXRs). However, COVID-19 manifestations can be accurately detected when present, motivating the use of these tools for evaluating disease progression on mild to severe COVID-19 patients.<\/jats:p>","DOI":"10.1038\/s41598-022-10568-3","type":"journal-article","created":{"date-parts":[[2022,4,21]],"date-time":"2022-04-21T18:06:50Z","timestamp":1650564410000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Assessing clinical applicability of COVID-19 detection in chest radiography with deep learning"],"prefix":"10.1038","volume":"12","author":[{"given":"Jo\u00e3o","family":"Pedrosa","sequence":"first","affiliation":[]},{"given":"Guilherme","family":"Aresta","sequence":"additional","affiliation":[]},{"given":"Carlos","family":"Ferreira","sequence":"additional","affiliation":[]},{"given":"Catarina","family":"Carvalho","sequence":"additional","affiliation":[]},{"given":"Joana","family":"Silva","sequence":"additional","affiliation":[]},{"given":"Pedro","family":"Sousa","sequence":"additional","affiliation":[]},{"given":"Lucas","family":"Ribeiro","sequence":"additional","affiliation":[]},{"given":"Ana Maria","family":"Mendon\u00e7a","sequence":"additional","affiliation":[]},{"given":"Aur\u00e9lio","family":"Campilho","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,4,21]]},"reference":[{"key":"10568_CR1","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1001\/jama.2020.12458","volume":"324","author":"M Klompas","year":"2020","unstructured":"Klompas, M., Baker, M. A. & Rhee, C. Airborne transmission of SARS-CoV-2: Theoretical considerations and available evidence. JAMA 324, 441\u2013442 (2020).","journal-title":"JAMA"},{"key":"10568_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12879-021-06528-3","volume":"21","author":"DA Mistry","year":"2021","unstructured":"Mistry, D. A., Wang, J. Y., Moeser, M.-E., Starkey, T. & Lee, L. Y. A systematic review of the sensitivity and specificity of lateral flow devices in the detection of sars-cov-2. BMC Infect. Dis. 21, 1\u201314 (2021).","journal-title":"BMC Infect. Dis."},{"key":"10568_CR3","unstructured":"SARS-CoV-2 Viral Mutations: Impact on COVID-19 Tests|FDA."},{"key":"10568_CR4","doi-asserted-by":"publisher","first-page":"101623","DOI":"10.1016\/j.tmaid.2020.101623","volume":"34","author":"AJ Rodriguez-Morales","year":"2020","unstructured":"Rodriguez-Morales, A. J. et al. Clinical, laboratory and imaging features of COVID-19: A systematic review and meta-analysis. Travel Med. Infect. Dis. 34, 101623 (2020).","journal-title":"Travel Med. Infect. Dis."},{"key":"10568_CR5","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1148\/radiol.2020201365","volume":"296","author":"GD Rubin","year":"2020","unstructured":"Rubin, G. D. et al. The role of chest imaging in patient management during the COVID-19 pandemic: A multinational consensus statement from the fleischner society. Radiology 296, 172\u2013180 (2020).","journal-title":"Radiology"},{"key":"10568_CR6","doi-asserted-by":"crossref","unstructured":"Cleverley, J., Piper, J. & Jones, M.\u00a0M. The role of chest radiography in confirming COVID-19 pneumonia. BMJ 370 (2020).","DOI":"10.1136\/bmj.m2426"},{"key":"10568_CR7","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1016\/j.radi.2014.02.007","volume":"20","author":"N Woznitza","year":"2014","unstructured":"Woznitza, N., Piper, K., Rowe, S. & West, C. Optimizing patient care in radiology through team-working: A case study from the united kingdom. Radiography 20, 258\u2013263 (2014).","journal-title":"Radiography"},{"key":"10568_CR8","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/RBME.2020.2987975","volume":"4","author":"F Shi","year":"2021","unstructured":"Shi, F. et\u00a0al. Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19. IEEE Rev. Biomed. Eng. 4, 4\u201314  (2021).","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"10568_CR9","unstructured":"Shoeibi, A. et\u00a0al. Automated detection and forecasting of COVID-19 using deep learning techniques: A review. arXiv preprint arXiv:2007.10785 (2020)."},{"key":"10568_CR10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-56847-4","volume":"10","author":"L Wang","year":"2020","unstructured":"Wang, L., Lin, Z. Q. & Wong, A. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest x-ray images. Sci. Rep. 10, 1\u201312 (2020).","journal-title":"Sci. Rep."},{"key":"10568_CR11","unstructured":"Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"10568_CR12","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1007\/s13246-020-00865-4","volume":"43","author":"I Apostolopoulos","year":"2020","unstructured":"Apostolopoulos, I.\u00a0D. & Mpesiana, T.\u00a0A. Covid-19: Automatic detection from x-ray images utilizing transfer learning with convolutional neural networks. Phys. Eng. Sci. Med. 43, 635\u2013640  (2020).","journal-title":"Phys. Eng. Sci. Med."},{"key":"10568_CR13","doi-asserted-by":"crossref","unstructured":"Redmon, J. & Farhadi, A. YOLO9000: Better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 7263\u20137271 (2017).","DOI":"10.1109\/CVPR.2017.690"},{"key":"10568_CR14","doi-asserted-by":"publisher","first-page":"103792","DOI":"10.1016\/j.compbiomed.2020.103792","volume":"121","author":"T Ozturk","year":"2020","unstructured":"Ozturk, T. et\u00a0al. Automated detection of COVID-19 cases using deep neural networks with x-ray images. Comput. Biol. Med. 121, 103792 (2020).","journal-title":"Comput. Biol. Med."},{"key":"10568_CR15","doi-asserted-by":"publisher","first-page":"e200337","DOI":"10.1148\/ryct.2020200337","volume":"2","author":"S Stephanie","year":"2020","unstructured":"Stephanie, S. et al. Determinants of chest x-ray sensitivity for COVID-19: A multi-institutional study in the United States. Radiol. Cardiothorac. Imag. 2, e200337 (2020).","journal-title":"Radiol. Cardiothorac. Imag."},{"key":"10568_CR16","unstructured":"Cohen, J.\u00a0P., Morrison, P. & Dao, L. Covid-19 image data collection. arXiv preprint arXiv:2003.11597 (2020)."},{"key":"10568_CR17","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1038\/s42256-021-00338-7","volume":"3","author":"AJ DeGrave","year":"2021","unstructured":"DeGrave, A. J., Janizek, J. D. & Lee, S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat. Mach. Intell. 3, 610\u2013619 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"10568_CR18","doi-asserted-by":"crossref","unstructured":"Irvin, J. et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33 590\u2013597 (2019).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"10568_CR19","doi-asserted-by":"crossref","unstructured":"Wang, X. et\u00a0al. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2097\u20132106 (2017).","DOI":"10.1109\/CVPR.2017.369"},{"key":"10568_CR20","unstructured":"Kaggle. RSNA pneumonia detection challenge|kaggle. https:\/\/www.kaggle.com\/c\/rsna-pneumonia-detection-challenge\/ (2018). Accessed 10 July 2020."},{"key":"10568_CR21","doi-asserted-by":"publisher","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","volume":"66","author":"A Bustos","year":"2020","unstructured":"Bustos, A., Pertusa, A., Salinas, J.-M. & de la Iglesia-Vay\u00e1, M. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020).","journal-title":"Med. Image Anal."},{"key":"10568_CR22","unstructured":"Vay\u00e1, M. d. l.\u00a0I. et\u00a0al. BIMCV COVID-19+: A large annotated dataset of RX and CT images from COVID-19 patients. arXiv preprint arXiv:2006.01174 (2020)."},{"key":"10568_CR23","doi-asserted-by":"crossref","unstructured":"Tabik, S. et\u00a0al. COVIDGR dataset and COVID-SDNet methodology for predicting COVID-19 based on chest x-ray images. arXiv preprint arXiv:2006.01409 (2020).","DOI":"10.1109\/JBHI.2020.3037127"},{"key":"10568_CR24","unstructured":"Redmon, J. & Farhadi, A. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)."},{"key":"10568_CR25","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 770\u2013778 (2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"10568_CR26","unstructured":"Bishop, C.\u00a0M. Pattern recognition and machine learning 235 (Springer, 2006)."},{"key":"10568_CR27","unstructured":"Kingma, D.\u00a0P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"10568_CR28","doi-asserted-by":"crossref","unstructured":"Deng, J. et\u00a0al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition 248\u2013255 (IEEE, 2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"10568_CR29","unstructured":"Lee, D.-H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation lLearning, ICML, Vol.\u00a03 (2013)."},{"key":"10568_CR30","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh, M. L. Interrater reliability: The kappa statistic. Biochem. Med. 22, 276\u2013282 (2012).","journal-title":"Biochem. Med."},{"key":"10568_CR31","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/BF02289261","volume":"13","author":"AL Edwards","year":"1948","unstructured":"Edwards, A. L. Note on the \u201ccorrection for continuity\u2019\u2019 in testing the significance of the difference between correlated proportions. Psychometrika 13, 185\u2013187 (1948).","journal-title":"Psychometrika"},{"key":"10568_CR32","doi-asserted-by":"crossref","unstructured":"Chattopadhay, A., Sarkar, A., Howlader, P. & Balasubramanian, V.\u00a0N. Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) 839\u2013847 (IEEE, 2018).","DOI":"10.1109\/WACV.2018.00097"},{"key":"10568_CR33","doi-asserted-by":"publisher","first-page":"3868","DOI":"10.1109\/TMI.2020.3006437","volume":"39","author":"A Mehrtash","year":"2020","unstructured":"Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P. & Kapur, T. Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Trans. Med. Imag. 39, 3868\u20133878. https:\/\/doi.org\/10.1109\/TMI.2020.3006437 (2020).","journal-title":"IEEE Trans. Med. Imag."},{"key":"10568_CR34","doi-asserted-by":"crossref","unstructured":"DeLong, E.\u00a0R., DeLong, D.\u00a0M. & Clarke-Pearson, D.\u00a0L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 837\u2013845 (1988).","DOI":"10.2307\/2531595"},{"key":"10568_CR35","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1111\/j.0006-341X.2000.01134.x","volume":"56","author":"E Venkatraman","year":"2000","unstructured":"Venkatraman, E. A permutation test to compare receiver operating characteristic curves. Biometrics 56, 1134\u20131138 (2000).","journal-title":"Biometrics"},{"key":"10568_CR36","doi-asserted-by":"crossref","unstructured":"Fisher, R.\u00a0A. Statistical methods for research workers. In Breakthroughs in statistics 66\u201370 (Springer, 1992).","DOI":"10.1007\/978-1-4612-4380-9_6"},{"key":"10568_CR37","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","volume":"56","author":"OJ Dunn","year":"1961","unstructured":"Dunn, O. J. Multiple comparisons among means. J. Am Stat. Assoc. 56, 52\u201364 (1961).","journal-title":"J. Am Stat. Assoc."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-10568-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-10568-3","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-10568-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T02:22:01Z","timestamp":1669342921000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-10568-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,21]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["10568"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-10568-3","relation":{},"ISSN":["2045-2322"],"issn-type":[{"type":"electronic","value":"2045-2322"}],"subject":[],"published":{"date-parts":[[2022,4,21]]},"assertion":[{"value":"20 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 April 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"6596"}}