{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T05:11:48Z","timestamp":1776057108587,"version":"3.50.1"},"reference-count":65,"publisher":"Springer Science and Business Media LLC","issue":"10","license":[{"start":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T00:00:00Z","timestamp":1665360000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T00:00:00Z","timestamp":1665360000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Saliency methods, which produce heat maps that highlight the areas of the medical image that influence model prediction, are often presented to clinicians as an aid in diagnostic decision-making. However, rigorous investigation of the accuracy and reliability of these strategies is necessary before they are integrated into the clinical setting. In this work, we quantitatively evaluate seven saliency methods, including Grad-CAM, across multiple neural network architectures using two evaluation metrics. We establish the first human benchmark for chest X-ray segmentation in a multilabel classification set-up, and examine under what clinical conditions saliency maps might be more prone to failure in localizing important pathologies compared with a human expert benchmark. We find that (1) while Grad-CAM generally localized pathologies better than the other evaluated saliency methods, all seven performed significantly worse compared with the human benchmark, (2) the gap in localization performance between Grad-CAM and the human benchmark was largest for pathologies that were smaller in size and had shapes that were more complex, and (3) model confidence was positively correlated with Grad-CAM localization performance. Our work demonstrates that several important limitations of saliency methods must be addressed before we can rely on them for deep learning explainability in medical imaging.<\/jats:p>","DOI":"10.1038\/s42256-022-00536-x","type":"journal-article","created":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T12:04:06Z","timestamp":1665403446000},"page":"867-878","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":193,"title":["Benchmarking saliency methods for chest X-ray interpretation"],"prefix":"10.1038","volume":"4","author":[{"given":"Adriel","family":"Saporta","sequence":"first","affiliation":[]},{"given":"Xiaotong","family":"Gui","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1301-2679","authenticated-orcid":false,"given":"Ashwin","family":"Agrawal","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1526-3685","authenticated-orcid":false,"given":"Anuj","family":"Pareek","sequence":"additional","affiliation":[]},{"given":"Steven Q. H.","family":"Truong","sequence":"additional","affiliation":[]},{"given":"Chanh D. T.","family":"Nguyen","sequence":"additional","affiliation":[]},{"given":"Van-Doan","family":"Ngo","sequence":"additional","affiliation":[]},{"given":"Jayne","family":"Seekins","sequence":"additional","affiliation":[]},{"given":"Francis G.","family":"Blankenberg","sequence":"additional","affiliation":[]},{"given":"Andrew Y.","family":"Ng","sequence":"additional","affiliation":[]},{"given":"Matthew P.","family":"Lungren","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8030-3727","authenticated-orcid":false,"given":"Pranav","family":"Rajpurkar","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,10,10]]},"reference":[{"key":"536_CR1","doi-asserted-by":"publisher","first-page":"e1002686","DOI":"10.1371\/journal.pmed.1002686","volume":"15","author":"P Rajpurkar","year":"2018","unstructured":"Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 15, e1002686 (2018).","journal-title":"PLoS Med."},{"key":"536_CR2","unstructured":"Rajpurkar, P. et al. CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. Preprint at: https:\/\/arxiv.org\/abs\/1711.05225 (2017)."},{"key":"536_CR3","doi-asserted-by":"publisher","first-page":"e1002699","DOI":"10.1371\/journal.pmed.1002699","volume":"15","author":"N Bien","year":"2018","unstructured":"Bien, N. et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Med. 15, e1002699 (2018).","journal-title":"PLoS Med."},{"key":"536_CR4","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1186\/s41747-020-00159-0","volume":"4","author":"G Baselli","year":"2020","unstructured":"Baselli, G., Codari, M. & Sardanelli, F. Opening the black box of machine learning in radiology: can the proximity of annotated cases be a way? Eur. Radiol. Exp. 4, 30 (2020).","journal-title":"Eur. Radiol. Exp."},{"key":"536_CR5","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","volume":"42","author":"G Litjens","year":"2017","unstructured":"Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60\u201388 (2017).","journal-title":"Med. Image Anal."},{"key":"536_CR6","doi-asserted-by":"publisher","first-page":"59","DOI":"10.7326\/M19-2548","volume":"172","author":"F Wang","year":"2019","unstructured":"Wang, F., Kaushal, R. & Khullar, D. Should health care demand interpretable artificial intelligence or accept \u2018black box\u2019 medicine? Ann. Intern. Med. 172, 59\u201360 (2019).","journal-title":"Ann. Intern. Med."},{"key":"536_CR7","first-page":"50","volume":"38","author":"B Goodman","year":"2017","unstructured":"Goodman, B. & Flaxman, S. European Union regulations on algorithmic decision-making and a \u2018right to explanation\u2019. AI Mag. 38, 50\u201357 (2017).","journal-title":"AI Mag."},{"key":"536_CR8","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-022-01806-2","volume":"46","author":"VK Venugopal","year":"2022","unstructured":"Venugopal, V. K., Takhar, R., Gupta, S., Saboo, A. & Mahajan, V. Clinical Explainability Failure (CEF) & Explainability Failure Ratio (EFR)\u2014changing the way we validate classification algorithms? J. Med. Syst. 46, 20 (2022).","journal-title":"J. Med. Syst."},{"key":"536_CR9","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-019-42557-4","volume":"9","author":"F Pasa","year":"2019","unstructured":"Pasa, F., Golkov, V., Pfeiffer, F., Cremers, D. & Pfeiffer, D. Efficient deep network architectures for fast chest X-ray tuberculosis screening and visualization. Sci. Rep. 9, 6268 (2019).","journal-title":"Sci. Rep."},{"key":"536_CR10","unstructured":"Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Workshop at International Conference on Learning Representations (2014)."},{"key":"536_CR11","unstructured":"Aggarwal, M. et al. Towards trainable saliency maps in medical imaging. Machine Learning for Health (ML4H) Extended Abstract Arxiv, Index:1\u20136 (2020)."},{"key":"536_CR12","unstructured":"Tjoa, E. & Guan, C. Quantifying explainability of saliency methods in deep neural networks. Preprint at: https:\/\/arxiv.org\/abs\/2009.02899 (2020)."},{"key":"536_CR13","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1038\/s41746-019-0105-1","volume":"2","author":"MA Badgeley","year":"2019","unstructured":"Badgeley, M. A. et al. Deep learning predicts hip fracture using confounding patient and healthcare variables. npj Digit. Med. 2, 31 (2019).","journal-title":"npj Digit. Med."},{"key":"536_CR14","doi-asserted-by":"publisher","first-page":"e1002683","DOI":"10.1371\/journal.pmed.1002683","volume":"15","author":"JR Zech","year":"2018","unstructured":"Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Med. 15, e1002683 (2018).","journal-title":"PLoS Med."},{"key":"536_CR15","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1038\/s42256-021-00338-7","volume":"3","author":"AJ DeGrave","year":"2021","unstructured":"DeGrave, A. J., Janizek, J. D. & Lee, S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat. Mach. Intell. 3, 610\u2013619 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"536_CR16","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-65105-x","volume":"10","author":"H Makimoto","year":"2020","unstructured":"Makimoto, H. et al. Performance of a convolutional neural network derived from an ECG database in recognizing myocardial infarction. Sci. Rep. 10, 8445 (2020).","journal-title":"Sci. Rep."},{"key":"536_CR17","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1038\/s41598-019-56927-5","volume":"10","author":"M Porumb","year":"2020","unstructured":"Porumb, M., Stranges, S., Pescap\u00e8, A. & Pecchia, L. Precision medicine and artificial intelligence: a pilot study on deep learning for hypoglycemic events detection based on ECG. Sci. Rep. 10, 170 (2020).","journal-title":"Sci. Rep."},{"key":"536_CR18","doi-asserted-by":"publisher","first-page":"e29","DOI":"10.1016\/S2589-7500(20)30271-5","volume":"3","author":"Y-C Tham","year":"2021","unstructured":"Tham, Y.-C. et al. Referral for disease-related visual impairment using retinal photograph-based deep learning: a proof-of-concept, model development study. Lancet Digit. Health 3, e29\u2013e40 (2021).","journal-title":"Lancet Digit. Health"},{"key":"536_CR19","doi-asserted-by":"publisher","first-page":"2861","DOI":"10.1167\/iovs.18-23887","volume":"59","author":"AV Varadarajan","year":"2018","unstructured":"Varadarajan, A. V. et al. Deep learning for predicting refractive error from retinal fundus images. Invest. Ophthalmol. Vis. Sci. 59, 2861\u20132868 (2018).","journal-title":"Invest. Ophthalmol. Vis. Sci."},{"key":"536_CR20","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1038\/s41551-019-0487-z","volume":"4","author":"A Mitani","year":"2020","unstructured":"Mitani, A. et al. Detection of anaemia from retinal fundus images via deep learning. Nat. Biomed. Eng. 4, 18\u201327 (2020).","journal-title":"Nat. Biomed. Eng."},{"key":"536_CR21","doi-asserted-by":"crossref","unstructured":"Lu, M. T. et al. Deep learning to assess long-term mortality from chest radiographs. JAMA Netw. Open 2, e197416 (2019).","DOI":"10.1001\/jamanetworkopen.2019.7416"},{"key":"536_CR22","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/s41746-020-00322-2","volume":"3","author":"P Rajpurkar","year":"2020","unstructured":"Rajpurkar, P. et al. CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV. npj Digit. Med. 3, 115 (2020).","journal-title":"npj Digit. Med."},{"key":"536_CR23","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-61055-6","volume":"10","author":"P Rajpurkar","year":"2020","unstructured":"Rajpurkar, P. et al. AppendiXNet: deep learning for diagnosis of appendicitis from a small dataset of CT exams using video pretraining. Sci. Rep. 10, 3958 (2020).","journal-title":"Sci. Rep."},{"key":"536_CR24","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1038\/s41746-020-0266-y","volume":"3","author":"S-C Huang","year":"2020","unstructured":"Huang, S.-C. et al. PENet\u2014a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging. npj Digit. Med. 3, 61 (2020).","journal-title":"npj Digit. Med."},{"key":"536_CR25","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","volume":"1","author":"C Rudin","year":"2019","unstructured":"Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206\u2013215 (2019).","journal-title":"Nat. Mach. Intell."},{"key":"536_CR26","doi-asserted-by":"crossref","unstructured":"Eitel, F. et al. Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer\u2019s disease classification. In Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support. ML-CDS IMIMIC 2019 (eds Suzuki, K. et al.) 3\u201311 (Lecture Notes in Computer Science Vol. 11797, Springer, 2019).","DOI":"10.1007\/978-3-030-33850-3_1"},{"key":"536_CR27","doi-asserted-by":"crossref","unstructured":"Young, K., et al. Deep neural network or dermatologist? In Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support. ML-CDS IMIMIC 2019 (eds Suzuki, K. et al.) 48\u201355 (Lecture Notes in Computer Science Vol. 11797, Springer, 2019).","DOI":"10.1007\/978-3-030-33850-3_6"},{"key":"536_CR28","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","volume":"3","author":"M Ghassemi","year":"2021","unstructured":"Ghassemi, M., Oakden-Rayner, L. & Beam, A. L. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit. Health 3, e745\u2013e750 (2021).","journal-title":"Lancet Digit. Health"},{"key":"536_CR29","doi-asserted-by":"publisher","first-page":"e190043","DOI":"10.1148\/ryai.2020190043","volume":"2","author":"M Reyes","year":"2020","unstructured":"Reyes, M. et al. On the interpretability of artificial intelligence in radiology: challenges and opportunities. Radiol. Artif. Intell. 2, e190043 (2020).","journal-title":"Radiol. Artif. Intell."},{"key":"536_CR30","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","volume":"128","author":"RR Selvaraju","year":"2020","unstructured":"Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis. 128, 336\u2013359 (2020).","journal-title":"Int. J. Comput. Vis."},{"key":"536_CR31","doi-asserted-by":"crossref","unstructured":"Chattopadhay, A., Sarkar, A., Howlader, P. & Balasubramanian, V. N. Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) 839\u2013847 (IEEE, 2018).","DOI":"10.1109\/WACV.2018.00097"},{"key":"536_CR32","unstructured":"Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. Proc. Mach. Learning Res. 70, 3319\u20133328 (2017)."},{"key":"536_CR33","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/s42979-021-00449-3","volume":"2","author":"M Bany Muhammad","year":"2021","unstructured":"Bany Muhammad, M. et al. Eigen-CAM: visual explanations for deep convolutional neural networks. SN Comput. Sci. 2, 47 (2021).","journal-title":"SN Comput. Sci."},{"key":"536_CR34","unstructured":"Shrikumar, A., Greenside, P. & Kundaje, A. Learning important features through propagating activation differences. Proc. Mach. Learning Res. 70, 3145\u20133153 (2017)."},{"key":"536_CR35","doi-asserted-by":"publisher","first-page":"e0130140","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach, S. et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10, e0130140 (2015).","journal-title":"PLoS ONE"},{"key":"536_CR36","doi-asserted-by":"crossref","unstructured":"Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision\u2014ECCV 2014 (eds Fleet, D. et al.) 818\u2013833 (Springer, 2014).","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"536_CR37","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2261\u20132269 (IEEE, 2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"536_CR38","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770\u2013778 (IEEE, 2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"536_CR39","doi-asserted-by":"crossref","unstructured":"Szegedy, C. et al. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1\u20139 (IEEE, 2015).","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"536_CR40","doi-asserted-by":"crossref","unstructured":"Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proc. AAAI Conference on Artificial Intelligence Vol. 33, 590\u2013597 (AAAI, 2019).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"536_CR41","doi-asserted-by":"crossref","unstructured":"Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9, 62\u201366 (1979).","DOI":"10.1109\/TSMC.1979.4310076"},{"key":"536_CR42","doi-asserted-by":"publisher","first-page":"1084","DOI":"10.1007\/s11263-017-1059-x","volume":"126","author":"J Zhang","year":"2018","unstructured":"Zhang, J. et al. Top-down neural attention by excitation backprop. Int. J. Comput. Vis. 126, 1084\u20131102 (2018).","journal-title":"Int. J. Comput. Vis."},{"key":"536_CR43","doi-asserted-by":"publisher","first-page":"e138","DOI":"10.1016\/S2589-7500(20)30003-0","volume":"2","author":"H-E Kim","year":"2020","unstructured":"Kim, H.-E. et al. Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digit. Health 2, e138\u2013e148 (2020).","journal-title":"Lancet Digit. Health"},{"key":"536_CR44","doi-asserted-by":"crossref","unstructured":"Efron, B. & Tibshirani, R. J. An Introduction to the Bootstrap (CRC Press, 1994).","DOI":"10.1201\/9780429246593"},{"key":"536_CR45","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-021-00915-w","volume":"8","author":"D Vrabac","year":"2021","unstructured":"Vrabac, D. et al. DLBCL-Morph: morphological features computed using deep learning for an annotated digital DLBCL image set. Sci. Data 8, 135 (2021).","journal-title":"Sci. Data"},{"key":"536_CR46","doi-asserted-by":"publisher","first-page":"102364","DOI":"10.1016\/j.media.2022.102364","volume":"77","author":"MS Ayhan","year":"2022","unstructured":"Ayhan, M. S. et al. Clinical validation of saliency maps for understanding deep neural networks in ophthalmology. Med. Image Anal. 77, 102364 (2022).","journal-title":"Med. Image Anal."},{"key":"536_CR47","doi-asserted-by":"publisher","first-page":"e200267","DOI":"10.1148\/ryai.2021200267","volume":"3","author":"N Arun","year":"2021","unstructured":"Arun, N. et al. Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging. Radiol. Artif. Intell. 3, e200267 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"536_CR48","doi-asserted-by":"crossref","unstructured":"Wang, X. et al. ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2097\u20132106 (IEEE, 2017).","DOI":"10.1109\/CVPR.2017.369"},{"key":"536_CR49","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1038\/s41597-022-01498-w","volume":"9","author":"HQ Nguyen","year":"2022","unstructured":"Nguyen, H. Q. et al. VinDr-CXR: An open dataset of chest X-rays with radiologist\u2019s annotations. Sci. Data 9, 429 (2022).","journal-title":"Sci. Data"},{"key":"536_CR50","unstructured":"Society for Imaging Informatics in Medicine (SIIM) SIIM-ACR pneumothorax segmentation. Kaggle https:\/\/kaggle.com\/c\/siim-acr-pneumothorax-segmentation (2019)."},{"key":"536_CR51","doi-asserted-by":"publisher","first-page":"1636","DOI":"10.1097\/PAS.0000000000001151","volume":"42","author":"DF Steiner","year":"2018","unstructured":"Steiner, D. F. et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am. J. Surg. Pathol. 42, 1636\u20131646 (2018).","journal-title":"Am. J. Surg. Pathol."},{"key":"536_CR52","unstructured":"Uyumazturk, B. et al. Deep learning for the digital pathologic diagnosis of cholangiocarcinoma and hepatocellular carcinoma: evaluating the impact of a web-based diagnostic assistant. Machine Learning for Health (ML4H) at NeurIPS - Extended Abstract (2019)."},{"key":"536_CR53","doi-asserted-by":"publisher","first-page":"e195600","DOI":"10.1001\/jamanetworkopen.2019.5600","volume":"2","author":"A Park","year":"2019","unstructured":"Park, A. et al. Deep learning-assisted diagnosis of cerebral aneurysms using the HeadXNet model. JAMA Netw. Open 2, e195600 (2019).","journal-title":"JAMA Netw. Open"},{"key":"536_CR54","unstructured":"Gadgil, S., Endo, M., Wen, E., Ng, A. Y. & Rajpurkar, P. CheXseg: combining expert annotations with DNN-generated saliency maps for X-ray segmentation. Proc. Mach. Learning Res. 143, 190\u2013204 (2021)."},{"key":"536_CR55","doi-asserted-by":"crossref","unstructured":"Crosby, J., Chen, S., Li, F., MacMahon, H. & Giger, M. Network output visualization to uncover limitations of deep learning detection of pneumothorax. Proc. SPIE 11316, 113160O (2020).","DOI":"10.1117\/12.2550066"},{"key":"536_CR56","first-page":"79","volume":"33","author":"H Melbye","year":"1992","unstructured":"Melbye, H. & Dale, K. Interobserver variability in the radiographic diagnosis of adult outpatient pneumonia. Acta Radiol. 33, 79\u201381 (1992).","journal-title":"Acta Radiol."},{"key":"536_CR57","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1378\/chest.68.3.278","volume":"68","author":"PG Herman","year":"1975","unstructured":"Herman, P. G. et al. Disagreements in chest Roentgen interpretation. CHEST 68, 278\u2013282 (1975).","journal-title":"CHEST"},{"key":"536_CR58","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1378\/chest.110.2.343","volume":"110","author":"MN Albaum","year":"1996","unstructured":"Albaum, M. N. et al. Interobserver reliability of the chest radiograph in community-acquired pneumonia. CHEST 110, 343\u2013350 (1996).","journal-title":"CHEST"},{"key":"536_CR59","unstructured":"Arun, N. T. et al. Assessing the validity of saliency maps for abnormality localization in medical imaging. In Tal Arbel, Ismail Ben Ayed, Marleen de Bruijne, Maxime Descoteaux, Herve Lombaert, Chris Pal (eds.), Medical Imaging with Deep Learning 2020, Short Paper Track (2020)."},{"key":"536_CR60","unstructured":"Graziani, M., Lompech, T., M\u00fcller, H. & Andrearczyk, V. Evaluation and comparison of CNN visual explanations for histopathology. In AAAI 2021, XAI Workshop (2021)."},{"key":"536_CR61","doi-asserted-by":"crossref","unstructured":"Choe, J. et al. Evaluating weakly supervised object localization methods right. In Proc. IEEE\/CVF Conference on Computer Vision and Pattern Recognition 3133\u20133142 (IEEE, 2020).","DOI":"10.1109\/CVPR42600.2020.00320"},{"key":"536_CR62","doi-asserted-by":"publisher","first-page":"e496","DOI":"10.1016\/S2589-7500(21)00106-0","volume":"3","author":"JCY Seah","year":"2021","unstructured":"Seah, J. C. Y. et al. Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study. Lancet Digit. Health 3, e496\u2013e506 (2021).","journal-title":"Lancet Digit. Health"},{"key":"536_CR63","unstructured":"Tan, M. & Le, Q. V. EfficientNet: rethinking model scaling for convolutional neural networks. Proc. Mach. Learning Res. 97, 6105\u20136114 (2019)."},{"key":"536_CR64","unstructured":"Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. International Conference on Learning Representations (ICLR) Poster (2015)."},{"key":"536_CR65","doi-asserted-by":"publisher","unstructured":"Saporta, A. et al. Code for \u2018Benchmarking saliency methods for chest X-ray interpretation\u2019. Zenodo https:\/\/doi.org\/10.5281\/zenodo.6973536 (2022).","DOI":"10.5281\/zenodo.6973536"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-022-00536-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-022-00536-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-022-00536-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,21]],"date-time":"2022-10-21T16:24:18Z","timestamp":1666369458000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-022-00536-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,10]]},"references-count":65,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["536"],"URL":"https:\/\/doi.org\/10.1038\/s42256-022-00536-x","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.02.28.21252634","asserted-by":"object"}]},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,10]]},"assertion":[{"value":"11 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 October 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"M.P.L. is an adviser for and\/or has research funded by GE, Philips, Carestream, Nines Radiology, Segmed, Centaur Labs, Microsoft, Bunkerhill and Amazon Web Services (none of the funded research was relevant to this project). A.P. is a medical associate at Cerebriu. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}