{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T09:17:53Z","timestamp":1784366273533,"version":"3.55.0"},"reference-count":47,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Deep learning, in recent times, has made remarkable strides when it comes to impressive performance for many tasks, including medical image processing. One of the contributing factors to these advancements is the emergence of large medical image datasets. However, it is exceedingly expensive and time-consuming to construct a large and trustworthy medical dataset; hence, there has been multiple research leveraging medical reports to automatically extract labels for data. The majority of this labor, however, is performed in English. In this work, we propose a data collecting and annotation pipeline that extracts information from Vietnamese radiology reports to provide accurate labels for chest X-ray (CXR) images. This can benefit Vietnamese radiologists and clinicians by annotating data that closely match their endemic diagnosis categories which may vary from country to country. To assess the efficacy of the proposed labeling technique, we built a CXR dataset containing 9,752 studies and evaluated our pipeline using a subset of this dataset. With an F1-score of at least 0.9923, the evaluation demonstrates that our labeling tool performs precisely and consistently across all classes. After building the dataset, we train deep learning models that leverage knowledge transferred from large public CXR datasets. We employ a variety of loss functions to overcome the curse of imbalanced multi-label datasets and conduct experiments with various model architectures to select the one that delivers the best performance. Our best model (CheXpert-pretrained EfficientNet-B2) yields an F1-score of 0.6989 (95% CI 0.6740, 0.7240), AUC of 0.7912, sensitivity of 0.7064 and specificity of 0.8760 for the abnormal diagnosis in general. Finally, we demonstrate that our coarse classification (based on five specific locations of abnormalities) yields comparable results to fine classification (twelve pathologies) on the benchmark CheXpert dataset for general anomaly detection while delivering better performance in terms of the average performance of all classes.<\/jats:p>","DOI":"10.1371\/journal.pone.0276545","type":"journal-article","created":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T17:38:43Z","timestamp":1667237923000},"page":"e0276545","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":4,"title":["Learning to diagnose common thorax diseases on chest radiographs from radiology reports in Vietnamese"],"prefix":"10.1371","volume":"17","author":[{"given":"Thao","family":"Nguyen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tam M.","family":"Vo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thang V.","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4851-2518","authenticated-orcid":true,"given":"Hieu H.","family":"Pham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ha Q.","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,10,31]]},"reference":[{"key":"pone.0276545.ref001","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/978-3-540-79942-9_2","volume-title":"Comparative interpretation of CT and standard radiography of the chest","author":"L Delrue","year":"2011"},{"key":"pone.0276545.ref002","unstructured":"American College of Radiology. ACR\u2013SPR\u2013STR practice parameter for the performance of chest radiography 2011; Available at: https:\/\/www.acr.org\/-\/media\/ACR\/Files\/Practice-Parameters\/ChestRad.pdf. Accessed August 22, 2021."},{"key":"pone.0276545.ref003","doi-asserted-by":"crossref","unstructured":"Tran, Thanh T and Pham, Hieu H and Nguyen, Thang V and Le, Tung T and Nguyen, Hieu T and Nguyen, Ha Q. Learning to automatically diagnose multiple diseases in pediatric chest radiographs using deep convolutional neural networks. Proceedings of the IEEE\/CVF International Conference on Computer Vision, (3314\u20133323), 2021.","DOI":"10.1101\/2021.08.12.21261954"},{"key":"pone.0276545.ref004","doi-asserted-by":"crossref","unstructured":"Pham, Hieu H and Nguyen, Ha Q and Nguyen, Hieu T and Le, Linh T and Khanh, Lam. An Accurate and Explainable Deep Learning System Improves Interobserver Agreement in the Interpretation of Chest Radiograph. arXiv:2208.03545, 2022.","DOI":"10.1101\/2021.09.28.21264286"},{"key":"pone.0276545.ref005","unstructured":"Le, Khiem H and Tran, Tuan V and Pham, Hieu H and Nguyen, Hieu T and Le, Tung T and Nguyen, Ha Q. Learning from Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis. arXiv preprint arXiv:2203.10611, 2022."},{"key":"pone.0276545.ref006","unstructured":"Rajpurkar, Pranav and Joshi, Anirudh and Pareek, Anuj and Chen, Phil and Kiani, Amirhossein and Irvin, Jeremy et al. CheXpedition: investigating generalization challenges for translation of chest x-ray algorithms to the clinical setting. arXiv preprint arXiv:2002.11379, 2020."},{"key":"pone.0276545.ref007","unstructured":"Wang, Hongyu and Xia, Yong. Chestnet: A deep neural network for classification of thoracic diseases on chest radiography. arXiv preprint arXiv:1807.03058, 2018."},{"key":"pone.0276545.ref008","article-title":"Automated abnormality classification of chest radiographs using deep convolutional neural networks","author":"Yu-Xing Tang","year":"2018","journal-title":"NPJ Digital Medicine (Nature Publishing Group)"},{"key":"pone.0276545.ref009","first-page":"104","article-title":"A transfer learning method with deep residual network for pediatric pneumonia diagnosis","author":"Gaobo Liang","year":"2020","journal-title":"Computer Methods and Programs in Biomedicine"},{"key":"pone.0276545.ref010","doi-asserted-by":"crossref","unstructured":"Nguyen, Hieu T and Pham, Hieu H and Nguyen, Nghia T and Nguyen, Ha Q and Huynh, Thang Q and Dao, Minh et al. VinDr-SpineXR: A deep learning framework for spinal lesions detection and classification from radiographs. International Conference on Medical Image Computing and Computer-Assisted Intervention, (pp. 291\u2013301), 2021.","DOI":"10.1007\/978-3-030-87240-3_28"},{"key":"pone.0276545.ref011","article-title":"Dicom imaging router: An open deep learning framework for classification of body parts from dicom x-ray scans","author":"Hieu H Pham","year":"2021","journal-title":"medRxiv"},{"key":"pone.0276545.ref012","doi-asserted-by":"crossref","unstructured":"Nguyen, Huyen TX and Tran, Sam B and Nguyen, Dung B and Pham, Hieu H and Nguyen, Ha Q. A novel multi-view deep learning approach for BI-RADS and density assessment of mammograms. arXiv preprint arXiv:2112.04490, 2021.","DOI":"10.1109\/EMBC48229.2022.9871564"},{"key":"pone.0276545.ref013","first-page":"1","article-title":"Deep learning to improve breast cancer detection on screening mammography","author":"Li Shen","year":"2019","journal-title":"Scientific Reports (Nature Publishing Group"},{"key":"pone.0276545.ref014","doi-asserted-by":"crossref","unstructured":"Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. InProceedings of the AAAI conference on artificial intelligence 2019 Jul 17 (Vol. 33, No. 01, pp. 590\u2013597).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"pone.0276545.ref015","doi-asserted-by":"crossref","unstructured":"Johnson AE, Pollard TJ, Greenbaum NR, Lungren MP, Deng CY, Peng Y, et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042. 2019 Jan 21.","DOI":"10.1038\/s41597-019-0322-0"},{"key":"pone.0276545.ref016","doi-asserted-by":"crossref","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","article-title":"PadChest: A large chest x-ray image dataset with multi-label annotated reports","volume":"66","author":"A Bustos","year":"2020","journal-title":"Med Image Anal"},{"key":"pone.0276545.ref017","doi-asserted-by":"crossref","unstructured":"Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. InProceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 2097\u20132106).","DOI":"10.1109\/CVPR.2017.369"},{"key":"pone.0276545.ref018","unstructured":"Nguyen, Hoang C and Le, Tung T and Pham, Hieu H and Nguyen, Ha Q. VinDr-RibCXR: A Benchmark Dataset for Automatic Segmentation and Labeling of Individual Ribs on Chest X-rays. arXiv preprint arXiv:2107.01327, 2021."},{"key":"pone.0276545.ref019","unstructured":"Nguyen HQ, Lam K, Le LT, Pham HH, Tran DQ, Nguyen DB, et al. VinDr-CXR: An open dataset of chest X-rays with radiologist\u2019s annotations. arXiv preprint arXiv:2012.15029. 2020 Dec 30."},{"key":"pone.0276545.ref020","doi-asserted-by":"crossref","unstructured":"Nguyen, H. Q., Pham, H. H., Tuan Linh, L., Dao, M., Khanh, L. VinDr-CXR: An open dataset of chest X-rays with radiologist annotations (version 1.0.0) PhysioNet, https:\/\/doi.org\/10.13026\/3akn-b287, 2021.","DOI":"10.1038\/s41597-022-01498-w"},{"key":"pone.0276545.ref021","doi-asserted-by":"crossref","unstructured":"Nguyen, Ngoc H and Pham, Hieu H and Tran, Thanh T and Nguyen, Tuan NM and Nguyen, Ha Q. VinDr-PCXR: An open, large-scale chest radiograph dataset for interpretation of common thoracic diseases in children. arXiv preprint arXiv:2203.10612, 2022.","DOI":"10.1101\/2022.03.04.22271937"},{"key":"pone.0276545.ref022","unstructured":"Pham, H. H., Tran, T. T., Nguyen, H. Q. VinDr-PCXR: An open, large-scale pediatric chest X-ray dataset for interpretation of common thoracic diseases. PhysioNet (version 1.0.0), https:\/\/doi.org\/10.13026\/k8qc-na36, 2022"},{"key":"pone.0276545.ref023","first-page":"188","article-title":"NegBio: a high-performance tool for negation and uncertainty detection in radiology reports","volume":"2018","author":"Y Peng","year":"2018","journal-title":"AMIA Summits on Translational Science Proceedings"},{"issue":"2","key":"pone.0276545.ref024","doi-asserted-by":"crossref","first-page":"490","DOI":"10.1007\/s10278-019-00299-9","article-title":"Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest X-ray dataset","volume":"33","author":"RW Filice","year":"2020","journal-title":"Journal of digital imaging"},{"key":"pone.0276545.ref025","unstructured":"Jain S, Agrawal A, Saporta A, Truong SQ, Bui T, Chambon P, et al. RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. Conference on Neural Information Processing Systems (NeurIPS 2021)"},{"key":"pone.0276545.ref026","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018 Oct 11."},{"key":"pone.0276545.ref027","unstructured":"McDermott MB, Hsu TM, Weng WH, Ghassemi M, Szolovits P. Chexpert++: Approximating the chexpert labeler for speed, differentiability, and probabilistic output. In Machine Learning for Healthcare Conference 2020 Sep 18 (pp. 913\u2013927). PMLR."},{"key":"pone.0276545.ref028","doi-asserted-by":"crossref","unstructured":"Jain S, Smit A, Truong SQ, Nguyen CD, Huynh MT, Jain M, et al. VisualCheXbert: addressing the discrepancy between radiology report labels and image labels. InProceedings of the Conference on Health, Inference, and Learning 2021 Apr 8 (pp. 105\u2013115).","DOI":"10.1145\/3450439.3451862"},{"key":"pone.0276545.ref029","doi-asserted-by":"crossref","unstructured":"Smit A, Jain S, Rajpurkar P, Pareek A, Ng AY, Lungren MP. CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. arXiv preprint arXiv:2004.09167. 2020 Apr 20.","DOI":"10.18653\/v1\/2020.emnlp-main.117"},{"issue":"1","key":"pone.0276545.ref030","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1017\/S1351324900000061","article-title":"Natural language processing in an operational clinical information system","volume":"1","author":"C Friedman","year":"1995","journal-title":"Natural Language Engineering"},{"issue":"5","key":"pone.0276545.ref031","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"GK Savova","year":"2010","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0276545.ref032","doi-asserted-by":"crossref","unstructured":"Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33; 2019. p. 590\u2013597.","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"pone.0276545.ref033","unstructured":"Rajpurkar P, Joshi A, Pareek A, Chen P, Kiani A, Irvin J, et al. CheXpedition: Investigating generalization challenges for translation of chest X-ray algorithms to the clinical setting. 2020. arXiv:2002.11379 [eess.IV]."},{"key":"pone.0276545.ref034","doi-asserted-by":"crossref","unstructured":"Pham HH, Le TT, Tran DQ, Ngo DT, Nguyen HQ. Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels. arXiv preprint arXiv:191106475. 2020.","DOI":"10.1101\/19013342"},{"issue":"1","key":"pone.0276545.ref035","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"AEW Johnson","year":"2019","journal-title":"Scientific Data"},{"issue":"11","key":"pone.0276545.ref036","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pmed.1002686","article-title":"Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists","volume":"15","author":"P Rajpurkar","year":"2018","journal-title":"PLoS Medicine"},{"issue":"2","key":"pone.0276545.ref037","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1148\/radiol.2019191293","article-title":"Chest radiograph interpretation with deep learning models: Assessment with radiologist-adjudicated reference standards and population-adjusted evaluation","volume":"294","author":"A Majkowska","year":"2020","journal-title":"Radiology"},{"key":"pone.0276545.ref038","doi-asserted-by":"crossref","unstructured":"Nguyen NH, Nguyen HQ, Nguyen NT, Nguyen TV, Pham HH, Nguyen TN. A clinical validation of VinDr-CXR, an AI system for detecting abnormal chest radiographs. arXiv preprint arXiv:2104.02256. 2021 Apr 6.","DOI":"10.3389\/fdgth.2022.890759"},{"key":"pone.0276545.ref039","doi-asserted-by":"crossref","unstructured":"Ke A, Ellsworth W, Banerjee O, Ng AY, Rajpurkar P. CheXtransfer: performance and parameter efficiency of ImageNet models for chest X-Ray interpretation. InProceedings of the Conference on Health, Inference, and Learning 2021 Apr 8 (pp. 116\u2013124).","DOI":"10.1145\/3450439.3451867"},{"key":"pone.0276545.ref040","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770\u2013778).","DOI":"10.1109\/CVPR.2016.90"},{"key":"pone.0276545.ref041","doi-asserted-by":"crossref","unstructured":"Huang G, Liu Z, Van Der Maaten L, Weinberger KQ. Densely connected convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 4700\u20134708).","DOI":"10.1109\/CVPR.2017.243"},{"key":"pone.0276545.ref042","doi-asserted-by":"crossref","unstructured":"Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 2818\u20132826).","DOI":"10.1109\/CVPR.2016.308"},{"key":"pone.0276545.ref043","unstructured":"Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks. InInternational Conference on Machine Learning 2019 May 24 (pp. 6105\u20136114). PMLR."},{"key":"pone.0276545.ref044","unstructured":"Ben-Baruch E, Ridnik T, Zamir N, Noy A, Friedman I, Protter M, et al. Asymmetric loss for multi-label classification. arXiv preprint arXiv:2009.14119. 2020 Sep 29."},{"key":"pone.0276545.ref045","doi-asserted-by":"crossref","unstructured":"Wu T, Huang Q, Liu Z, Wang Y, Lin D. Distribution-balanced loss for multi-label classification in long-tailed datasets. In European Conference on Computer Vision 2020 Aug 23 (pp. 162\u2013178). Springer, Cham.","DOI":"10.1007\/978-3-030-58548-8_10"},{"key":"pone.0276545.ref046","unstructured":"Efron B, Tibshirani RJ. An Introduction to the Bootstrap. No. 57 in Monographs on Statistics and Applied Probability. Boca Raton, Florida, USA: Chapman & Hall\/CRC; 1993."},{"key":"pone.0276545.ref047","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision 2017 (pp. 618\u2013626).","DOI":"10.1109\/ICCV.2017.74"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0276545","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T17:39:24Z","timestamp":1667237964000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0276545"}},"subtitle":[],"editor":[{"given":"Tarik A.","family":"Rashid","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":47,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10,31]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0276545","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,31]]}}}