{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T23:51:37Z","timestamp":1790725897032,"version":"4.1.0"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,1,17]],"date-time":"2022-01-17T00:00:00Z","timestamp":1642377600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,17]],"date-time":"2022-01-17T00:00:00Z","timestamp":1642377600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Australian Research Council\u0393\u00c7\u00d6s Discovery Projects funding scheme","award":["Project DP190102181 and DP210101465"],"award-info":[{"award-number":["Project DP190102181 and DP210101465"]}]},{"name":"Australian Research Council\u0393\u00c7\u00d6s Discovery Projects funding scheme","award":["Project DP190102181 and DP210101465"],"award-info":[{"award-number":["Project DP190102181 and DP210101465"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Deep neural networks (DNNs) have been widely applied for detecting COVID-19 in medical images. Existing studies mainly apply transfer learning and other data representation strategies to generate accurate point estimates. The generalization power of these networks is always questionable due to being developed using small datasets and failing to report their predictive confidence. Quantifying uncertainties associated with DNN predictions is a prerequisite for their trusted deployment in medical settings. Here we apply and evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray (CXR) images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced. Through comprehensive experiments, it is shown that networks pertained on CXR images outperform networks pretrained on natural image datasets such as ImageNet. Qualitatively and quantitatively evaluations also reveal that the predictive uncertainty estimates are statistically higher for erroneous predictions than correct predictions. Accordingly, uncertainty quantification methods are capable of flagging risky predictions with high uncertainty estimates. We also observe that ensemble methods more reliably capture uncertainties during the inference. DNN-based solutions for COVID-19 detection have been mainly proposed without any principled mechanism for risk mitigation. Previous studies have mainly focused on on generating single-valued predictions using pretrained DNNs. In this paper, we comprehensively apply and comparatively evaluate three uncertainty quantification techniques for COVID-19 detection using chest X-Ray images. The novel concept of uncertainty confusion matrix is proposed and new performance metrics for the objective evaluation of uncertainty estimates are introduced for the first time. Using these new uncertainty performance metrics, we quantitatively demonstrate when we could trust DNN predictions for COVID-19 detection from chest X-rays. It is important to note the proposed novel uncertainty evaluation metrics are generic and could be applied for evaluation of probabilistic forecasts in all classification problems.<\/jats:p>","DOI":"10.1038\/s41598-022-05052-x","type":"journal-article","created":{"date-parts":[[2022,1,17]],"date-time":"2022-01-17T06:03:22Z","timestamp":1642399402000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":62,"title":["Objective evaluation of deep uncertainty predictions for COVID-19 detection"],"prefix":"10.1038","volume":"12","author":[{"given":"Hamzeh","family":"Asgharnezhad","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Afshar","family":"Shamsi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roohallah","family":"Alizadehsani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abbas","family":"Khosravi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saeid","family":"Nahavandi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zahra Alizadeh","family":"Sani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dipti","family":"Srinivasan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sheikh Mohammed Shariful","family":"Islam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,1,17]]},"reference":[{"key":"5052_CR1","doi-asserted-by":"crossref","unstructured":"Zu, Z. Y., Jiang, M. D., Xu, P. P., Chen, W., Ni, Q. Q., Lu, G. M. & Zhang, L. J. Coronavirus disease 2019 (covid-19): A perspective from China. Radiology, 200490 (2020).","DOI":"10.1148\/radiol.2020200490"},{"issue":"7639","key":"5052_CR2","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115\u2013118 (2017).","journal-title":"Nature"},{"issue":"1","key":"5052_CR3","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","volume":"25","author":"A Esteva","year":"2019","unstructured":"Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25(1), 24\u201329 (2019).","journal-title":"Nat. Med."},{"key":"5052_CR4","doi-asserted-by":"crossref","unstructured":"Minaee, S., Kafieh, R., Sonka, M., Yazdani, S. & Soufi, G. J. Deep-covid: Predicting covid-19 from chest x-ray images using deep transfer learning. arXiv preprintarXiv:2004.09363 (2020).","DOI":"10.1016\/j.media.2020.101794"},{"key":"5052_CR5","unstructured":"Shoeibi, A., Khodatars, M., Alizadehsani, R., Ghassemi, N., Jafari, M., Moridian, P., Khadem, A., Sadeghi, D., Hussain, S., Zare, A. et\u00a0al., Automated detection and forecasting of covid-19 using deep learning techniques: A review. arXiv preprintarXiv:2007.10785 (2020)."},{"key":"5052_CR6","doi-asserted-by":"crossref","unstructured":"Lalmuanawma, S., Hussain, J. & Chhakchhuak, L. Applications of machine learning and artificial intelligence for covid-19 (sars-cov-2) pandemic: A review. Chaos Solitons Fractals, 110059 (2020).","DOI":"10.1016\/j.chaos.2020.110059"},{"key":"5052_CR7","doi-asserted-by":"crossref","unstructured":"Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y. & Shen, D. Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19. IEEE Rev. Biomed. Eng. 14, 4\u201315 (2020).","DOI":"10.1109\/RBME.2020.2987975"},{"key":"5052_CR8","doi-asserted-by":"crossref","unstructured":"Hanumanthu, S.\u00a0Rekha. Role of intelligent computing in covid-19 prognosis: A state-of-the-art review. Chaos Solitons Fractals, 109947 (2020).","DOI":"10.1016\/j.chaos.2020.109947"},{"key":"5052_CR9","doi-asserted-by":"crossref","unstructured":"Daniel, T. M. Toman\u2019s tuberculosis. case detection, treatment, and monitoring. questions and answers. Am. J. Trop. Med. Hygiene 73(1), 229 (2005).","DOI":"10.4269\/ajtmh.2005.73.229"},{"key":"5052_CR10","unstructured":"Raghu, M.,\u00a0Blumer, K.,\u00a0Sayres, R.,\u00a0Obermeyer, Z.,\u00a0Kleinberg, B.,\u00a0Mullainathan, S.,&\u00a0Kleinberg, J. Direct uncertainty prediction for medical second opinions. In International Conference on Machine Learning, pp.\u00a05281\u20135290 (2019)."},{"key":"5052_CR11","unstructured":"Mukhoti, J.&\u00a0Gal, Y. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprintarXiv:1811.12709 (2018)."},{"key":"5052_CR12","doi-asserted-by":"crossref","unstructured":"Subedar, M.,\u00a0Krishnan, R., Meyer, P.\u00a0L.,\u00a0Tickoo, O.&\u00a0Huang, J. Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEE International Conference on Computer Vision, pp.\u00a06301\u20136310 (2019).","DOI":"10.1109\/ICCV.2019.00640"},{"key":"5052_CR13","volume-title":"Bayesian Theory","author":"JM Bernardo","year":"2009","unstructured":"Bernardo, J. M. & Smith, A. F. Bayesian Theory Vol. 405 (Wiley, London, 2009)."},{"key":"5052_CR14","unstructured":"Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp.\u00a02348\u20132356 (2011)."},{"key":"5052_CR15","unstructured":"Blundell, C.,\u00a0Cornebise, J.,\u00a0Kavukcuoglu, K.&\u00a0Wierstra, D. Weight uncertainty in neural networks. arXiv preprintarXiv:1505.05424 (2015)."},{"key":"5052_CR16","unstructured":"Gal, Y.&\u00a0Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp.\u00a01050\u20131059 (2016)."},{"key":"5052_CR17","unstructured":"Lakshminarayanan, B.,\u00a0Pritzel, A.&\u00a0Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp.\u00a06402\u20136413 (2017)."},{"key":"5052_CR18","doi-asserted-by":"crossref","unstructured":"Choi, S.,\u00a0Lee, K.,\u00a0Lim, S.&\u00a0Oh, S. Uncertainty-aware learning from demonstration using mixture density networks with sampling-free variance modeling. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp.\u00a06915\u20136922. IEEE (2018).","DOI":"10.1109\/ICRA.2018.8462978"},{"key":"5052_CR19","unstructured":"van Amersfoort, J.,\u00a0Smith, L., Teh, Y.\u00a0W.&\u00a0Gal, Y. Uncertainty estimation using a single deep deterministic neural network."},{"key":"5052_CR20","unstructured":"Goodfellow, I.,\u00a0Pouget-Abadie, J.,\u00a0Mirza, M.,\u00a0Xu, B.,\u00a0Warde-Farley, D.,\u00a0Ozair, S.,\u00a0Courville, A.&\u00a0Bengio, Y. Generative adversarial nets. Adv. Neural Inform. Process. Syst. 27 (2014)."},{"issue":"2","key":"5052_CR21","doi-asserted-by":"publisher","first-page":"84","DOI":"10.26599\/BDMA.2020.9020012","volume":"4","author":"KK Singh","year":"2021","unstructured":"Singh, K. K. & Singh, A. Diagnosis of covid-19 from chest x-ray images using wavelets-based depthwise convolution network. Big Data Min. Anal. 4(2), 84\u201393 (2021).","journal-title":"Big Data Min. Anal."},{"issue":"5","key":"5052_CR22","doi-asserted-by":"publisher","first-page":"759","DOI":"10.26599\/TST.2021.9010026","volume":"26","author":"J Pang","year":"2021","unstructured":"Pang, J., Huang, Y., Xie, Z., Li, J. & Cai, Z. Collaborative city digital twin for the covid-19 pandemic: A federated learning solution. Tsinghua Sci. Technol. 26(5), 759\u2013771 (2021).","journal-title":"Tsinghua Sci. Technol."},{"issue":"2","key":"5052_CR23","doi-asserted-by":"publisher","first-page":"65","DOI":"10.26599\/BDMA.2020.9020013","volume":"4","author":"R Kumari","year":"2021","unstructured":"Kumari, R. et al. Analysis and predictions of spread, recovery, and death caused by covid-19 in India. Big Data Min. Anal. 4(2), 65\u201375 (2021).","journal-title":"Big Data Min. Anal."},{"key":"5052_CR24","doi-asserted-by":"publisher","first-page":"105372","DOI":"10.1016\/j.cnsns.2020.105372","volume":"90","author":"T Alberti","year":"2020","unstructured":"Alberti, T. & Faranda, D. On the uncertainty of real-time predictions of epidemic growths: A covid-19 case study for china and Italy. Commun. Nonlinear Sci. Numer. Simul. 90, 105372 (2020).","journal-title":"Commun. Nonlinear Sci. Numer. Simul."},{"key":"5052_CR25","doi-asserted-by":"crossref","unstructured":"Ghoshal, B.,\u00a0Tucker, A.,\u00a0Sanghera, B. & Wong, W.\u00a0L. Estimating uncertainty in deep learning for reporting confidence to clinicians when segmenting nuclei image data. In 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS), pp.\u00a0318\u2013324. IEEE (2019).","DOI":"10.1109\/CBMS.2019.00072"},{"key":"5052_CR26","unstructured":"Ghoshal, B. & Tucker, A. Estimating uncertainty and interpretability in deep learning for coronavirus (covid-19) detection. arXiv preprintarXiv:2003.10769 (2020)."},{"key":"5052_CR27","doi-asserted-by":"crossref","unstructured":"Ghoshal, B. & Tucker, A. On cost-sensitive calibrated uncertainty in deep learning: An application on covid-19 detection. In 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), pp.\u00a0503\u2013509. IEEE (2021).","DOI":"10.1109\/CBMS52027.2021.00100"},{"key":"5052_CR28","unstructured":"Jokandan, A.\u00a0S., Asgharnezhad, H., Jokandan, S.\u00a0S., Khosravi, A., Kebria, P.\u00a0M.,\u00a0Nahavandi, D.,\u00a0Nahavandi, S.&\u00a0Srinivasan, D. An uncertainty-aware transfer learning-based framework for covid-19 diagnosis. arXiv preprintarXiv:2007.14846 (2020)."},{"key":"5052_CR29","doi-asserted-by":"crossref","unstructured":"Alizadehsani, R.,\u00a0Sharifrazi, D., Izadi, N.\u00a0H., Joloudari, J.\u00a0H.,\u00a0Shoeibi, A., Gorriz, J.\u00a0M.,\u00a0Hussain, S., Arco, J.\u00a0E., Sani, Z.\u00a0A.,\u00a0Khozeimeh, F. et\u00a0al., Uncertainty-aware semi-supervised method using large unlabelled and limited labeled covid-19 data. arXiv preprintarXiv:2102.06388 (2021).","DOI":"10.1145\/3462635"},{"key":"5052_CR30","doi-asserted-by":"crossref","unstructured":"Calderon-Ramirez, S., Yang, S., Moemeni, A., Colreavy-Donnelly, S., Elizondo, D.\u00a0A., Oala, L., Rodr\u00edguez-Capit\u00e1n, J., Jim\u00e9nez-Navarro, M., L\u00f3pez-Rubio, E. & Molina-Cabello, M.\u00a0A. Improving uncertainty estimation with semi-supervised deep learning for covid-19 detection using chest x-ray images. IEEE Access (2021).","DOI":"10.1109\/ACCESS.2021.3085418"},{"key":"5052_CR31","doi-asserted-by":"crossref","unstructured":"Hoebel, K., Andrearczyk, V., Beers, A., Patel, J., Chang, K., Depeursinge, A.,\u00a0M\u00fcller, H. & Kalpathy-Cramer, J. An exploration of uncertainty information for segmentation quality assessment. In Medical Imaging 2020: Image Processing, vol.\u00a011313, p.\u00a0113131K, International Society for Optics and Photonics (2020).","DOI":"10.1117\/12.2548722"},{"key":"5052_CR32","unstructured":"Mallick, A., Dwivedi, C., Kailkhura, B., Joshi, G. & Han, T.\u00a0Y.-J. Can your ai differentiate cats from covid-19? sample efficient uncertainty estimation for deep learning safety. Choice, 50, 6."},{"issue":"2","key":"5052_CR33","doi-asserted-by":"publisher","first-page":"227","DOI":"10.26599\/TST.2019.9010007","volume":"25","author":"Y Jin","year":"2019","unstructured":"Jin, Y., Guo, W. & Zhang, Y. A time-aware dynamic service quality prediction approach for services. Tsinghua Sci. Technol. 25(2), 227\u2013238 (2019).","journal-title":"Tsinghua Sci. Technol."},{"key":"5052_CR34","unstructured":"van Amersfoort, J.,\u00a0Smith, L., Teh, Y.\u00a0W.&\u00a0Gal, Y. Simple and scalable epistemic uncertainty estimation using a single deep deterministic neural network. arXiv preprintarXiv:2003.02037 (2020)."},{"key":"5052_CR35","doi-asserted-by":"crossref","unstructured":"Cohen, J.\u00a0P.,\u00a0Dao, L.,\u00a0Morrison, P.,\u00a0Roth, K.,\u00a0Bengio, Y.,\u00a0Shen, B.,\u00a0Abbasi, A.,\u00a0Hoshmand-Kochi, M.,\u00a0Ghassemi, M.,\u00a0Li, H. et\u00a0al. Predicting covid-19 pneumonia severity on chest x-ray with deep learning. arXiv preprintarXiv:2005.11856 (2020).","DOI":"10.7759\/cureus.9448"},{"issue":"1","key":"5052_CR36","doi-asserted-by":"publisher","first-page":"e180041","DOI":"10.1148\/ryai.2019180041","volume":"1","author":"G Shih","year":"2019","unstructured":"Shih, G. et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia. Radiol. Artif. Intell. 1(1), e180041 (2019).","journal-title":"Radiol. Artif. Intell."},{"key":"5052_CR37","first-page":"590","volume":"33","author":"J Irvin","year":"2019","unstructured":"Irvin, J. et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proc. AAAI Conf. Artif. Intell. 33, 590\u2013597 (2019).","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"5052_CR38","doi-asserted-by":"crossref","unstructured":"Wang, X.,\u00a0Peng, Y.,\u00a0Lu, L.,\u00a0Lu, Z.,\u00a0Bagheri, M. & Summers, R.\u00a0M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\u00a02097\u20132106, (2017).","DOI":"10.1109\/CVPR.2017.369"},{"issue":"2","key":"5052_CR39","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1148\/radiol.2019191293","volume":"294","author":"A Majkowska","year":"2020","unstructured":"Majkowska, A. et al. Chest radiograph interpretation with deep learning models: Assessment with radiologist-adjudicated reference standards and population-adjusted evaluation. Radiology 294(2), 421\u2013431 (2020).","journal-title":"Radiology"},{"key":"5052_CR40","doi-asserted-by":"crossref","unstructured":"Johnson, A.\u00a0E., Pollard, T.\u00a0J., Greenbaum, N.\u00a0R., Lungren, M.\u00a0P., Deng, C.-y.,\u00a0Peng, Y.,\u00a0Lu, Z., Mark, R.\u00a0G., Berkowitz, S.\u00a0J.&\u00a0Horng, S. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprintarXiv:1901.07042 (2019).","DOI":"10.1038\/s41597-019-0322-0"},{"key":"5052_CR41","doi-asserted-by":"crossref","unstructured":"Bustos, A.,\u00a0Pertusa, A., Salinas, J.-M.&\u00a0de\u00a0la Iglesia-Vay\u00e1, M. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Med. Image Anal. 101797 (2020).","DOI":"10.1016\/j.media.2020.101797"},{"issue":"2","key":"5052_CR42","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1093\/jamia\/ocv080","volume":"23","author":"D Demner-Fushman","year":"2016","unstructured":"Demner-Fushman, D. et al. Preparing a collection of radiology examinations for distribution and retrieval. J. Am. Med. Inform. Assoc. 23(2), 304\u2013310 (2016).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"5052_CR43","unstructured":"Raghu, M.,\u00a0Zhang, C.,\u00a0Kleinberg, J.&\u00a0Bengio, S. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems, pp.\u00a03347\u20133357 (2019)."},{"key":"5052_CR44","doi-asserted-by":"crossref","unstructured":"Huang, G.,\u00a0Liu, Z.,\u00a0Van Der\u00a0Maaten, L. & Weinberger, K.\u00a0Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\u00a04700\u20134708 (2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"5052_CR45","doi-asserted-by":"crossref","unstructured":"Islam, M.,\u00a0Karray, F.,\u00a0Alhajj, R.,\u00a0Zeng, J. et\u00a0al. A review on deep learning techniques for the diagnosis of novel coronavirus (covid-19). arXiv preprintarXiv:2008.04815 (2020).","DOI":"10.1109\/ACCESS.2021.3058537"},{"key":"5052_CR46","unstructured":"Guo, C., Pleiss, G., Sun, Y. & Weinberger, K.\u00a0Q. On calibration of modern neural networks. arXiv preprintarXiv:1706.04599 (2017)."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05052-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05052-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05052-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T17:09:13Z","timestamp":1669309753000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-05052-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,17]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["5052"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-05052-x","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-890026\/v1","asserted-by":"object"}]},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,17]]},"assertion":[{"value":"9 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 January 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"815"}}