{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,25]],"date-time":"2026-10-25T03:54:34Z","timestamp":1792900474204,"version":"4.5.0"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,6]],"date-time":"2022-04-06T00:00:00Z","timestamp":1649203200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,6]],"date-time":"2022-04-06T00:00:00Z","timestamp":1649203200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Commun"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The inability to accurately, efficiently label large, open-access medical imaging datasets limits the widespread implementation of artificial intelligence models in healthcare. There have been few attempts, however, to automate the annotation of such public databases; one approach, for example, focused on labor-intensive, manual labeling of subsets of these datasets to be used to train new models. In this study, we describe a method for standardized, automated labeling based on similarity to a previously validated, explainable AI (xAI) model-derived-atlas, for which the user can specify a quantitative threshold for a desired level of accuracy (the probability-of-similarity, pSim metric). We show that our xAI model, by calculating the pSim values for each clinical output label based on comparison to its training-set derived reference atlas, can automatically label the external datasets to a user-selected, high level of accuracy, equaling or exceeding that of human experts. We additionally show that, by fine-tuning the original model using the automatically labelled exams for retraining, performance can be preserved or improved, resulting in a highly accurate, more generalized model.<\/jats:p>","DOI":"10.1038\/s41467-022-29437-8","type":"journal-article","created":{"date-parts":[[2022,4,6]],"date-time":"2022-04-06T06:08:13Z","timestamp":1649225293000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["Accurate auto-labeling of chest X-ray images based on quantitative similarity to an explainable AI model"],"prefix":"10.1038","volume":"13","author":[{"given":"Doyun","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1572-8056","authenticated-orcid":false,"given":"Joowon","family":"Chung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jongmun","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc D.","family":"Succi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Conklin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria Gabriela Figueiro","family":"Longo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5022-9530","authenticated-orcid":false,"given":"Jeanne B.","family":"Ackman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brent P.","family":"Little","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Milena","family":"Petranovic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9938-7476","authenticated-orcid":false,"given":"Mannudeep K.","family":"Kalra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0236-7319","authenticated-orcid":false,"given":"Michael H.","family":"Lev","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6211-7050","authenticated-orcid":false,"given":"Synho","family":"Do","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,4,6]]},"reference":[{"key":"29437_CR1","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1038\/s41551-018-0324-9","volume":"3","author":"H Lee","year":"2019","unstructured":"Lee, H. et al. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat. Biomed. Eng. 3, 173\u2013182 (2019).","journal-title":"Nat. Biomed. Eng."},{"key":"29437_CR2","doi-asserted-by":"crossref","unstructured":"Irvin, J. et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence 33, 590\u2013597 (2019).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"29437_CR3","doi-asserted-by":"publisher","unstructured":"Johnson, A., et al. MIMIC-CXR-JPG - chest radiographs with structured labels (version 2.0.0). PhysioNet https:\/\/doi.org\/10.13026\/8360-t248 (2019).","DOI":"10.13026\/8360-t248"},{"key":"29437_CR4","doi-asserted-by":"crossref","unstructured":"Wang, X., et al. Chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2097\u20132106 (2017).","DOI":"10.1109\/CVPR.2017.369"},{"key":"29437_CR5","doi-asserted-by":"publisher","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","volume":"66","author":"A Bustos","year":"2020","unstructured":"Bustos, A., Pertusa, A., Salinas, J. M. & de la Iglesia-Vay\u00e1, M. Padchest: a large chest x-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020).","journal-title":"Med. Image Anal."},{"key":"29437_CR6","doi-asserted-by":"publisher","first-page":"5078","DOI":"10.21037\/jtd.2019.08.34","volume":"12","author":"TK Kim","year":"2020","unstructured":"Kim, T. K., Paul, H. Y., Hager, G. D. & Lin, C. T. Refining dataset curation methods for deep learning-based automated tuberculosis screening. J. Thorac. Dis. 12, 5078\u20135085 (2020).","journal-title":"J. Thorac. Dis."},{"key":"29437_CR7","doi-asserted-by":"crossref","unstructured":"Witowski, J., et al. MarkIt: a collaborative artificial intelligence annotation platform leveraging blockchain for medical imaging research. Blockchain in Healthcare Today (2021).","DOI":"10.30953\/bhty.v4.176"},{"key":"29437_CR8","first-page":"37","volume":"2","author":"D Powers","year":"2008","unstructured":"Powers, D. Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation. J. Mach. Learn. Technol. 2, 37\u201363 (2008).","journal-title":"J. Mach. Learn. Technol."},{"key":"29437_CR9","doi-asserted-by":"publisher","first-page":"e200267","DOI":"10.1148\/ryai.2021200267","volume":"3","author":"N Arun","year":"2021","unstructured":"Arun, N. et al. Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radiol. Artif. Intell. 3, e200267 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"29437_CR10","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770\u2013778, (2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"29437_CR11","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C. Mobilenetv2: inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4510\u20134520, (2018).","DOI":"10.1109\/CVPR.2018.00474"},{"key":"29437_CR12","doi-asserted-by":"crossref","unstructured":"Tan, M., et al Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 2820\u20132828, (2019).","DOI":"10.1109\/CVPR.2019.00293"},{"key":"29437_CR13","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1148\/radiol.2017162326","volume":"284","author":"P Lakhani","year":"2017","unstructured":"Lakhani, P. & Sundaram, B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284, 574\u2013582 (2017).","journal-title":"Radiology"},{"key":"29437_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-019-42294-8","volume":"9","author":"IM Baltruschat","year":"2019","unstructured":"Baltruschat, I. M., Nickisch, H., Grass, M., Knopp, T. & Saalbach, A. Comparison of deep learning approaches for multi-label chest X-ray classification. Sci. Rep. 9, 1\u201310 (2019).","journal-title":"Sci. Rep."},{"key":"29437_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-019-42557-4","volume":"9","author":"F Pasa","year":"2019","unstructured":"Pasa, F., Golkov, V., Pfeiffer, F., Cremers, D. & Pfeiffer, D. Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization. Sci. Rep. 9, 1\u20139 (2019).","journal-title":"Sci. Rep."},{"key":"29437_CR16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-56847-4","volume":"10","author":"L Wang","year":"2020","unstructured":"Wang, L., Lin, Z. Q. & Wong, A. Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Sci. Rep. 10, 1\u201312 (2020).","journal-title":"Sci. Rep."},{"key":"29437_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00322-2","volume":"3","author":"P Rajpurkar","year":"2020","unstructured":"Rajpurkar, P. et al. CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV. NPJ Digit. Med. 3, 1\u20138 (2020).","journal-title":"NPJ Digit. Med."},{"key":"29437_CR18","doi-asserted-by":"publisher","first-page":"2688","DOI":"10.1109\/TMI.2020.2993291","volume":"39","author":"Y Oh","year":"2020","unstructured":"Oh, Y., Park, S. & Ye, J. C. Deep learning covid-19 features on cxr using limited training data sets. IEEE Trans. Med. Imaging 39, 2688\u20132700 (2020).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"29437_CR19","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1148\/radiol.2018180237","volume":"290","author":"JG Nam","year":"2019","unstructured":"Nam, J. G. et al. Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 290, 218\u2013228 (2019).","journal-title":"Radiology"},{"key":"29437_CR20","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1148\/radiol.2019182465","volume":"294","author":"Y Sim","year":"2020","unstructured":"Sim, Y. et al. Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs. Radiology 294, 199\u2013209 (2020).","journal-title":"Radiology"},{"key":"29437_CR21","doi-asserted-by":"crossref","unstructured":"Sung, J., et al. Added value of deep learning-based detection system for multiple major findings on chest radiographs: a randomized crossover study. Radiology 202818, (2021).","DOI":"10.1148\/radiol.2021202818"},{"key":"29437_CR22","doi-asserted-by":"publisher","first-page":"e1002683","DOI":"10.1371\/journal.pmed.1002683","volume":"15","author":"JR Zech","year":"2018","unstructured":"Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 15, e1002683 (2018).","journal-title":"PLoS Med."},{"key":"29437_CR23","doi-asserted-by":"publisher","first-page":"e200152","DOI":"10.1148\/ryai.2021200152","volume":"4","author":"AM Rauschecker","year":"2021","unstructured":"Rauschecker, A. M. et al. Interinstitutional portability of a deep learning brain MRI lesion segmentation algorithm. Radiol. Artif. Intell. 4, e200152 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"29437_CR24","doi-asserted-by":"publisher","unstructured":"McInnes, L. et al. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 3, 861 https:\/\/doi.org\/10.21105\/joss.00861 (2018).","DOI":"10.21105\/joss.00861"},{"key":"29437_CR25","unstructured":"Berthelot, D., et al. Mixmatch: a holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, 5050\u20135060 (2019)."},{"key":"29437_CR26","doi-asserted-by":"crossref","unstructured":"He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 9729\u20139738, (2020).","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"29437_CR27","unstructured":"Chen, T., Kornblith, S., Norouzi, M. and Hinton, G. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning, 1597\u20131607, (2020)."},{"key":"29437_CR28","unstructured":"Caron, M., et al. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), (2020)."},{"key":"29437_CR29","doi-asserted-by":"crossref","unstructured":"Hadsell, R., Chopra, S. and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1735\u20131742, (2006).","DOI":"10.1109\/CVPR.2006.100"},{"key":"29437_CR30","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1007\/s13246-020-00865-4","volume":"43","author":"ID Apostolopoulos","year":"2020","unstructured":"Apostolopoulos, I. D. & Mpesiana, T. A. Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks. Phys. Eng. Sci. Med. 43, 635\u2013640 (2020).","journal-title":"Phys. Eng. Sci. Med."},{"key":"29437_CR31","doi-asserted-by":"publisher","first-page":"1285","DOI":"10.1109\/TMI.2016.2528162","volume":"35","author":"HC Shin","year":"2016","unstructured":"Shin, H. C. et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285\u20131298 (2016).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"29437_CR32","unstructured":"Yosinski, J. et al. How transferable are features in deep neural networks? Advances in neural information processing systems 27 (2014)."},{"key":"29437_CR33","unstructured":"Kolb, D. A. Experiential Learning: Experience As the Source of Learning and Development (FT press, 2014)."},{"key":"29437_CR34","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L. and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700\u20134708, (2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"29437_CR35","doi-asserted-by":"crossref","unstructured":"Deng, J., et al. Imagenet: a large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 248\u2013255, (2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"29437_CR36","unstructured":"Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)."},{"key":"29437_CR37","unstructured":"Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, (2019)."},{"key":"29437_CR38","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2921\u20132929, (2016).","DOI":"10.1109\/CVPR.2016.319"},{"key":"29437_CR39","doi-asserted-by":"crossref","unstructured":"Selvaraju, R. R., et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, 618\u2013626, (2017).","DOI":"10.1109\/ICCV.2017.74"},{"key":"29437_CR40","doi-asserted-by":"publisher","first-page":"2800","DOI":"10.1080\/02664763.2018.1441383","volume":"45","author":"C Ju","year":"2018","unstructured":"Ju, C., Bibaut, A. & van der Laan, M. The relative performance of ensemble methods with deep convolutional neural networks for image classification. J. Appl. Stat. 45, 2800\u20132818 (2018).","journal-title":"J. Appl. Stat."},{"key":"29437_CR41","doi-asserted-by":"crossref","unstructured":"Ahn, J., Cho, S. and Kwak, S. Weakly supervised learning of instance segmentation with inter-pixel relations. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 2209\u20132218, (2019).","DOI":"10.1109\/CVPR.2019.00231"},{"key":"29437_CR42","doi-asserted-by":"crossref","unstructured":"Jo, S. & Yu, I. J. Puzzle-CAM: improved localization via matching partial and full features. 2021 IEEE International Conference on Image Processing, (2021).","DOI":"10.1109\/ICIP42928.2021.9506058"},{"key":"29437_CR43","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1016\/S0034-4257(97)00083-7","volume":"62","author":"SV Stehman","year":"1997","unstructured":"Stehman, S. V. Selecting and interpreting measures of thematic classification accuracy. Remote Sens. Environ. 62, 77\u201389 (1997).","journal-title":"Remote Sens. Environ."}],"container-title":["Nature Communications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-29437-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-29437-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-29437-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,21]],"date-time":"2024-09-21T13:50:20Z","timestamp":1726926620000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-29437-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,6]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["29437"],"URL":"https:\/\/doi.org\/10.1038\/s41467-022-29437-8","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-830690\/v1","asserted-by":"object"}]},"ISSN":["2041-1723"],"issn-type":[{"value":"2041-1723","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,6]]},"assertion":[{"value":"20 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"M.H.L. is a consultant for GE Healthcare and for the Takeda, Roche, and Seagen Pharmaceutical Companies, and has received institutional research support from Siemens Healthcare. B.P.L. and J.B.A. receive royalties from Elsevier, Inc. as an associate academic textbook editor and author. S.D. is a consultant of Doai and received research support from Tplus and Medibloc. M.K.K. has received institutional research support from Siemens Healthineers, Coreline Inc., and Riverain Tech Inc. J.M.C. was partially supported by a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI) funded by the Ministry of Health & Welfare, Republic of Korea (HI19C1057). The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"1867"}}