{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T09:44:15Z","timestamp":1753868655607,"version":"3.41.2"},"reference-count":58,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2024,4,8]],"date-time":"2024-04-08T00:00:00Z","timestamp":1712534400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Numer Methods Biomed Eng"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Several data sets have been collected and various artificial intelligence models have been developed for COVID\u201019 classification and detection from both chest radiography (CXR) and thorax computed tomography (CTX) images. However, the pitfalls and shortcomings of these systems significantly limit their clinical use. In this respect, improving the weaknesses of advanced models can be very effective besides developing new ones. The inability to diagnose ground\u2010glass opacities by conventional CXR has limited the use of this modality in the diagnostic work\u2010up of COVID\u201019. In our study, we investigated whether we could increase the diagnostic efficiency by collecting a novel CXR data set, which contains pneumonic regions that are not visible to the experts and can only be annotated under CTX guidance. We develop an ensemble methodology of well\u2010established deep CXR models for this new data set and develop a machine learning\u2010based non\u2010maximum suppression strategy to boost the performance for challenging CXR images. CTX and CXR images of 379 patients who applied to our hospital with suspected COVID\u201019 were evaluated with consensus by seven radiologists. Among these, CXR images of 161 patients who also have had a CTX examination on the same day or until the day before or after and whose CTX findings are compatible with COVID\u201019 pneumonia, are selected for annotating. CTX images are arranged in the main section passing through the anterior, middle, and posterior according to the sagittal plane with the reformed maximum intensity projection (MIP) method in the coronal plane. Based on the analysis of coronal MIP reconstructed CTX images, the regions corresponding to the pneumonia foci are annotated manually in CXR images. Radiologically classified posterior to anterior (PA) CXR of 218 patients with negative thorax CTX imaging were classified as COVID\u201019 pneumonia negative group. Accordingly, we have collected a new data set using anonymized CXR (JPEG) and CT (DICOM) images, where the PA CXRs contain pneumonic regions that are hidden or not easily recognized and annotated under CTX guidance. The reference finding was the presence of pneumonic infiltration consistent with COVID\u201019 on chest CTX examination. COVID\u2010Net, a specially designed convolutional neural network, was used to detect cases of COVID\u201019 among CXRs. Diagnostic performances were evaluated by ROC analysis by applying six COVID\u2010Net variants (COVIDNet\u2010CXR3\u2010A, \u2010B, \u2010C\/COVIDNet\u2010CXR4\u2010A, \u2010B, \u2010C) to the defined data set and combining these models in various ways via ensemble strategies. Finally, a convex optimization strategy is carried out to find the outperforming weighted ensemble of individual models. The mean age of 161 patients with pneumonia was 49.31\u2009\u00b1\u200915.12, and the median age was 48\u2009years. The mean age of 218 patients without signs of pneumonia in thorax CTX examination was 40.04\u2009\u00b1\u200914.46, and the median was 38. When working with different combinations of COVID\u2010Net's six variants, the area under the curve (AUC) using the ensemble COVID\u2010Net CXR 4A\u20104B\u20103C was .78, sensitivity 67%, specificity 95%; COVID\u2010Net CXR 4a\u20103b\u20103c was .79, sensitivity 69% and specificity 94%. When diverse and complementary COVID\u2010Net models are used together through an ensemble, it has been determined that the AUC values are close to other studies, and the specificity is significantly higher than other studies in the literature.<\/jats:p>","DOI":"10.1002\/cnm.3823","type":"journal-article","created":{"date-parts":[[2024,4,8]],"date-time":"2024-04-08T09:23:39Z","timestamp":1712568219000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Thorax computed tomography (CTX) guided ground truth annotation of CHEST radiographs (CXR) for improved classification and detection of COVID\u201019"],"prefix":"10.1002","volume":"40","author":[{"given":"\u015e\u00fckr\u00fc Mehmet","family":"Ert\u00fcrk","sequence":"first","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tu\u011f\u00e7e","family":"Toprak","sequence":"additional","affiliation":[{"name":"Institute of Natural and Applied Sciences Dokuz Eylul University  \u0130zmir Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rana G\u00fcn\u00f6z","family":"C\u00f6mert","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cemre","family":"Candemir","sequence":"additional","affiliation":[{"name":"International Computer Institute Ege University  Bornova Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eda","family":"Cing\u00f6z","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zeynep Nur","family":"Akyol Sari","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2922-5847","authenticated-orcid":false,"given":"Celal Caner","family":"Ercan","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Esin","family":"D\u00fcvek","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0161-1914","authenticated-orcid":false,"given":"Berke","family":"Ersoy","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edanur","family":"Karapinar","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Atadan","family":"Tunaci","sequence":"additional","affiliation":[{"name":"Radiodiagnostics Department Istanbul University, Istanbul Faculty of Medicine  Istanbul Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M. Alper","family":"Selver","sequence":"additional","affiliation":[{"name":"Electrical and Electronics Engineering Department Dokuz Eylul University, Faculty of Engineering  \u0130zmir Turkey"},{"name":"Izmir Health Technologies Development and Accelerator (BioIzmir) Dokuz Eylul University  \u0130zmir Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2024,4,8]]},"reference":[{"key":"e_1_2_11_2_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41568\u2010018\u20100016\u20105"},{"key":"e_1_2_11_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591\u2010020\u20100931\u20103"},{"key":"e_1_2_11_4_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2018.00034"},{"key":"e_1_2_11_5_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocv080"},{"key":"e_1_2_11_6_1","doi-asserted-by":"crossref","unstructured":"IrvinJ RajpurkarP KoM et al.CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.arXiv.2019. doi:10.48550\/arXiv.1901.07031","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"e_1_2_11_7_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41597\u2010019\u20100322\u20100"},{"key":"e_1_2_11_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.369"},{"key":"e_1_2_11_9_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2015151169"},{"key":"e_1_2_11_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101797"},{"key":"e_1_2_11_11_1","doi-asserted-by":"crossref","unstructured":"SmitA JainS RajpurkarP PareekA NgAY LungrenMP.CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT.arXiv.2020. doi:10.48550\/arXiv.2004.09167","DOI":"10.18653\/v1\/2020.emnlp-main.117"},{"key":"e_1_2_11_12_1","first-page":"1305","article-title":"AI accelerated human\u2010in\u2010the\u2010loop structuring of radiology reports","volume":"2020","author":"Wu JT","year":"2020","journal-title":"AMIA Annu Symp Proc AMIA Symp"},{"key":"e_1_2_11_13_1","first-page":"1537","volume-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems","author":"Li CY","year":"2018"},{"key":"e_1_2_11_14_1","first-page":"249","volume-title":"Proceedings of the 4th Machine Learning for Healthcare Conference","author":"Liu G","year":"2019"},{"key":"e_1_2_11_15_1","doi-asserted-by":"crossref","unstructured":"WangX PengY LuL LuZ SummersRM.TieNet: Text\u2010Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X\u2010rays.arXiv.2018. doi:10.48550\/arXiv.1801.04334","DOI":"10.1109\/CVPR.2018.00943"},{"key":"e_1_2_11_16_1","unstructured":"WuJT AguNN LourentzouI et al.Chest ImaGenome Dataset for Clinical Reasoning.arXiv.2021. doi:10.48550\/arXiv.2108.00316"},{"key":"e_1_2_11_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/s0197\u20102456(00)00097\u20100"},{"key":"e_1_2_11_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dib.2020.106056"},{"key":"e_1_2_11_19_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256\u2010021\u201000307\u20100"},{"key":"e_1_2_11_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10278\u2010019\u201000299\u20109"},{"key":"e_1_2_11_21_1","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2019180041"},{"key":"e_1_2_11_22_1","doi-asserted-by":"publisher","DOI":"10.3978\/j.issn.2223\u20104292.2014.11.20"},{"key":"e_1_2_11_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2021.3069169"},{"key":"e_1_2_11_24_1","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph17186933"},{"key":"e_1_2_11_25_1","unstructured":"GhoshalB TuckerA.Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID\u201019) Detection.arXiv.2020. doi:10.48550\/arXiv.2003.10769"},{"key":"e_1_2_11_26_1","doi-asserted-by":"crossref","unstructured":"BarariaA GhoshA BoseC BharD.Network for subclinical prognostication of COVID 19 Patients from data of thoracic roentgenogram: A feasible alternative screening technology.medRxiv. p. 2020.09.07.201898522020. doi:10.1101\/2020.09.07.20189852","DOI":"10.1101\/2020.09.07.20189852"},{"key":"e_1_2_11_27_1","unstructured":"FarooqM HafeezA.COVID\u2010ResNet: A Deep Learning Framework for Screening of COVID19 from Radiographs.arXiv.2020. doi:10.48550\/arXiv.2003.14395"},{"key":"e_1_2_11_28_1","doi-asserted-by":"crossref","unstructured":"KanaEBG KanaMGZ KanaAFD KenfackRHA.A web\u2010based Diagnostic Tool for COVID\u201019 Using Machine Learning on Chest Radiographs (CXR). medRxiv. p. 2020.04.21.200632632020. doi:10.1101\/2020.04.21.20063263","DOI":"10.1101\/2020.04.21.20063263"},{"key":"e_1_2_11_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12539\u2010020\u201000393\u20105"},{"key":"e_1_2_11_30_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2020202944"},{"key":"e_1_2_11_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2020.106742"},{"key":"e_1_2_11_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s42600\u2010021\u201000132\u20109"},{"key":"e_1_2_11_33_1","doi-asserted-by":"crossref","unstructured":"LiX LiC ZhuD.COVID\u2010MobileXpert: On\u2010Device COVID\u201019 Patient Triage and Follow\u2010up using Chest X\u2010rays.arXiv.2020. doi:10.48550\/arXiv.2004.03042","DOI":"10.1109\/BIBM49941.2020.9313217"},{"key":"e_1_2_11_34_1","doi-asserted-by":"publisher","DOI":"10.3233\/XST\u2010200715"},{"key":"e_1_2_11_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2020.104284"},{"key":"e_1_2_11_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3466690"},{"key":"e_1_2_11_37_1","doi-asserted-by":"publisher","DOI":"10.3892\/etm.2020.8797"},{"key":"e_1_2_11_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_2_11_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s42600\u2010021\u201000151\u20106"},{"key":"e_1_2_11_40_1","unstructured":"TanM LeQV.EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.arXiv.2020. doi:10.48550\/arXiv.1905.11946"},{"key":"e_1_2_11_41_1","unstructured":"MalhotraA MittalS MajumdarP et al.Multi\u2010Task Driven Explainable Diagnosis of COVID\u201019 using Chest X\u2010ray Images.arXiv.2020. doi:10.48550\/arXiv.2008.03205"},{"key":"e_1_2_11_42_1","unstructured":"SayyedAQMS SahaD HossainAR.CovMUNET: A Multiple Loss Approach towards Detection of COVID\u201019 from Chest X\u2010ray.arXiv.2020. doi:10.48550\/arXiv.2007.14318"},{"key":"e_1_2_11_43_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598\u2010020\u201076550\u2010z"},{"key":"e_1_2_11_44_1","unstructured":"WangL.COVID\u2010Net Open Source Initiative.2023. Accessed March 23 2023. Availablehttps:\/\/github.com\/lindawangg\/COVID-Net\/blob\/65999806546b55ab8adfc32b0cf8f70880c337dc\/docs\/models.md"},{"key":"e_1_2_11_45_1","unstructured":"CohenJP MorrisonP DaoL.COVID\u201019 image data collection.arXiv:2003.11597.2020."},{"key":"e_1_2_11_46_1","unstructured":"Radiological Society of North America.RSNA Pneumonia Detection Challenge.2019https:\/\/www.kaggle.com\/c\/rsna-pneumonia-detection-challenge\/data"},{"key":"e_1_2_11_47_1","unstructured":"ChungA.Figure 1 COVID\u201019 chest x\u2010ray data initiative.2020https:\/\/github.com\/agchung\/Figure1-COVID-chestxray-dataset"},{"key":"e_1_2_11_48_1","unstructured":"ChungA.Actualmed COVID\u201019 chest x\u2010ray data initiative.2020https:\/\/github.com\/agchung\/Actualmed-COVID-chestxray-dataset"},{"key":"e_1_2_11_49_1","unstructured":"Radiological Society of North America.COVID\u201019 Radiography Database.2019https:\/\/www.kaggle.com\/tawsifurrahman\/covid19radiographydatabase"},{"key":"e_1_2_11_50_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598\u2010021\u201088538\u20104"},{"key":"e_1_2_11_51_1","doi-asserted-by":"publisher","DOI":"10.5152\/dir.2019.20294"},{"key":"e_1_2_11_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/RBME.2019.2946868"},{"key":"e_1_2_11_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_2_11_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.685"},{"key":"e_1_2_11_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/AICCSA50499.2020.9316491"},{"key":"e_1_2_11_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC55140.2022.9922411"},{"key":"e_1_2_11_57_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2020201874"},{"key":"e_1_2_11_58_1","doi-asserted-by":"publisher","DOI":"10.1186\/s41747\u2010020\u201000203\u2010z"},{"key":"e_1_2_11_59_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2019.108774"}],"container-title":["International Journal for Numerical Methods in Biomedical Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cnm.3823","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,10]],"date-time":"2024-06-10T12:18:54Z","timestamp":1718021934000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/cnm.3823"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,8]]},"references-count":58,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1002\/cnm.3823"],"URL":"https:\/\/doi.org\/10.1002\/cnm.3823","archive":["Portico"],"relation":{},"ISSN":["2040-7939","2040-7947"],"issn-type":[{"type":"print","value":"2040-7939"},{"type":"electronic","value":"2040-7947"}],"subject":[],"published":{"date-parts":[[2024,4,8]]},"assertion":[{"value":"2023-03-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-27","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e3823"}}