{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T04:44:58Z","timestamp":1781325898758,"version":"3.54.1"},"reference-count":79,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,7,30]],"date-time":"2022-07-30T00:00:00Z","timestamp":1659139200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,30]],"date-time":"2022-07-30T00:00:00Z","timestamp":1659139200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>While many deep-learning-based computer-aided detection systems (CAD) have been developed and commercialized for abnormality detection in chest radiographs (CXR), their ability to localize a target abnormality is rarely reported. Localization accuracy is important in terms of model interpretability, which is crucial in clinical settings. Moreover, diagnostic performances are likely to vary depending on thresholds which define an accurate localization. In a multi-center, stand-alone clinical trial using temporal and external validation datasets of 1,050 CXRs, we evaluated localization accuracy, localization-adjusted discrimination, and calibration of a commercially available deep-learning-based CAD for detecting consolidation and pneumothorax. The CAD achieved image-level AUROC (95% CI) of 0.960 (0.945, 0.975), sensitivity of 0.933 (0.899, 0.959), specificity of 0.948 (0.930, 0.963), dice of 0.691 (0.664, 0.718), moderate calibration for consolidation, and image-level AUROC of 0.978 (0.965, 0.991), sensitivity of 0.956 (0.923, 0.978), specificity of 0.996 (0.989, 0.999), dice of 0.798 (0.770, 0.826), moderate calibration for pneumothorax. Diagnostic performances varied substantially when localization accuracy was accounted for but remained high at the minimum threshold of clinical relevance. In a separate trial for diagnostic impact using 461 CXRs, the causal effect of the CAD assistance on clinicians\u2019 diagnostic performances was estimated. After adjusting for age, sex, dataset, and abnormality type, the CAD improved clinicians\u2019 diagnostic performances on average (OR [95% CI]\u2009=\u20091.73 [1.30, 2.32]; <jats:italic>p<\/jats:italic>\u2009&lt;\u20090.001), although the effects varied substantially by clinical backgrounds. The CAD was found to have high stand-alone diagnostic performances and may beneficially impact clinicians\u2019 diagnostic performances when used in clinical settings.<\/jats:p>","DOI":"10.1038\/s41746-022-00658-x","type":"journal-article","created":{"date-parts":[[2022,7,30]],"date-time":"2022-07-30T06:03:04Z","timestamp":1659160984000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Localization-adjusted diagnostic performance and assistance effect of a computer-aided detection system for pneumothorax and consolidation"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9332-049X","authenticated-orcid":false,"given":"Sun Yeop","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangwoo","family":"Ha","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min Gyeong","family":"Jeon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4341-0643","authenticated-orcid":false,"given":"Hao","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyunju","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hwa Pyung","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ye Ra","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hoseok","family":"I","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yeon Joo","family":"Jeong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoon Ha","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyemin","family":"Ahn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sang Hyup","family":"Hong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyun Jung","family":"Koo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Choong Wook","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min Jae","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yeon Joo","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyung Won","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3267-8989","authenticated-orcid":false,"given":"Jong Mun","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,30]]},"reference":[{"key":"658_CR1","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1016\/S0009-9260(97)80280-2","volume":"52","author":"GR Tudor","year":"1997","unstructured":"Tudor, G. R., Finlay, D. & Taub, N. An assessment of inter-observer agreement and accuracy when reporting plain radiographs. Clin. Radiol. 52, 235\u2013238 (1997).","journal-title":"Clin. Radiol."},{"key":"658_CR2","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1378\/chest.110.2.343","volume":"110","author":"MN Albaum","year":"1996","unstructured":"Albaum, M. N. et al. Interobserver reliability of the chest radiograph in community-acquired pneumonia. Chest 110, 343\u2013350 (1996).","journal-title":"Chest"},{"key":"658_CR3","unstructured":"World Health Organization. Chest Radiography in Tuberculosis Detection - Summary of current WHO recommendations and guidance on programmatic approaches. (2016)."},{"key":"658_CR4","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1148\/radiology.217.2.r00nv14456","volume":"217","author":"EJ Potchen","year":"2000","unstructured":"Potchen, E. J. et al. Measuring performance in chest radiography. Radiology 217, 456\u2013459 (2000).","journal-title":"Radiology"},{"key":"658_CR5","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1378\/chest.10-2946","volume":"140","author":"W Ding","year":"2011","unstructured":"Ding, W., Shen, Y., Yang, J., He, X. & Zhang, M. Diagnosis of pneumothorax by radiography and ultrasonography: A meta-analysis. Chest 140, 859\u2013866 (2011).","journal-title":"Chest"},{"key":"658_CR6","doi-asserted-by":"publisher","first-page":"e007838","DOI":"10.1136\/bmjopen-2015-007838","volume":"5","author":"M Hew","year":"2015","unstructured":"Hew, M., Corcoran, J. P., Harriss, E. K., Rahman, N. M. & Mallett, S. The diagnostic accuracy of chest ultrasound for CT-detected radiographic consolidation in hospitalised adults with acute respiratory failure: A systematic review. BMJ Open 5, e007838 (2015).","journal-title":"BMJ Open"},{"key":"658_CR7","doi-asserted-by":"publisher","DOI":"10.1186\/cc13016","volume":"17","author":"S Alrajab","year":"2013","unstructured":"Alrajab, S., Youssef, A. M., Akkus, N. I. & Caldito, G. Pleural ultrasonography versus chest radiography for the diagnosis of pneumothorax: Review of the literature and meta-analysis. Crit. Care 17, R208 (2013).","journal-title":"Crit. Care"},{"key":"658_CR8","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1016\/j.jphys.2020.12.002","volume":"67","author":"L Hansell","year":"2021","unstructured":"Hansell, L., Milross, M., Delaney, A., Tian, D. H. & Ntoumenopoulos, G. Lung ultrasound has greater accuracy than conventional respiratory assessment tools for the diagnosis of pleural effusion, lung consolidation and collapse: A systematic review. J. Physiother. 67, 41\u201348 (2021).","journal-title":"J. Physiother."},{"key":"658_CR9","first-page":"29","volume":"13","author":"A Ebrahimi","year":"2014","unstructured":"Ebrahimi, A. et al. Diagnostic accuracy of chest ultrasonography versus chest radiography for identification of pneumothorax: A systematic review and meta-analysis. Tanaffos 13, 29\u201340 (2014).","journal-title":"Tanaffos"},{"key":"658_CR10","doi-asserted-by":"publisher","first-page":"e707","DOI":"10.1097\/CCM.0000000000003129","volume":"46","author":"MH Winkler","year":"2018","unstructured":"Winkler, M. H., Touw, H. R., van de Ven, P. M., Twisk, J. & Tuinman, P. R. Diagnostic accuracy of chest radiograph, and when concomitantly studied lung ultrasound, in critically Ill patients with respiratory symptoms: A systematic review and meta-analysis. Crit. Care Med. 46, e707\u2013e714 (2018).","journal-title":"Crit. Care Med."},{"key":"658_CR11","doi-asserted-by":"publisher","first-page":"101034","DOI":"10.1016\/j.eclinm.2021.101034","volume":"38","author":"G Frija","year":"2021","unstructured":"Frija, G. et al. How to improve access to medical imaging in low- and middle-income countries? EClinical Med. 38, 101034 (2021).","journal-title":"EClinical Med."},{"key":"658_CR12","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1148\/radiol.2020201434","volume":"297","author":"DJ Mollura","year":"2020","unstructured":"Mollura, D. J. et al. Artificial intelligence in low- and middle-income countries: Innovating global health radiology. Radiology 297, 513\u2013520 (2020).","journal-title":"Radiology"},{"key":"658_CR13","doi-asserted-by":"crossref","unstructured":"World Health Organization. WHO consolidated guidelines on tuberculosis. Module 2: screening - systematic screening for tuberculosis disease. (2021).","DOI":"10.30978\/TB2021-2-86"},{"key":"658_CR14","doi-asserted-by":"publisher","first-page":"102125","DOI":"10.1016\/j.media.2021.102125","volume":"72","author":"E \u00c7all\u0131","year":"2021","unstructured":"\u00c7all\u0131, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G. & Murphy, K. Deep learning for chest X-ray analysis: A survey. Med. Image Anal. 72, 102125 (2021).","journal-title":"Med. Image Anal."},{"key":"658_CR15","doi-asserted-by":"publisher","first-page":"511","DOI":"10.3348\/kjr.2019.0821","volume":"21","author":"EJ Hwang","year":"2020","unstructured":"Hwang, E. J. & Park, C. M. Clinical implementation of deep learning in thoracic radiology: Potential applications and challenges. Korean J. Radiol. 21, 511\u2013525 (2020).","journal-title":"Korean J. Radiol."},{"key":"658_CR16","unstructured":"Rajpurkar, P. et al. CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)."},{"key":"658_CR17","doi-asserted-by":"crossref","unstructured":"Thian, Y. L. et al. Deep learning systems for pneumothorax detection on chest radiographs: A multicenter external validation study. Radiol. Artif. Intell. 3 (2021).","DOI":"10.1148\/ryai.2021200190"},{"key":"658_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00322-2","volume":"3","author":"P Rajpurkar","year":"2020","unstructured":"Rajpurkar, P. et al. CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV. npj Digit. Med. 3, 1\u20138 (2020).","journal-title":"npj Digit. Med."},{"key":"658_CR19","doi-asserted-by":"publisher","first-page":"e2017135","DOI":"10.1001\/jamanetworkopen.2020.17135","volume":"3","author":"H Yoo","year":"2020","unstructured":"Yoo, H., Kim, K. H., Singh, R., Digumarthy, S. R. & Kalra, M. K. Validation of a deep learning algorithm for the detection of malignant pulmonary nodules in chest radiographs. JAMA Netw. Open 3, e2017135 (2020).","journal-title":"JAMA Netw. Open"},{"key":"658_CR20","doi-asserted-by":"publisher","first-page":"e496","DOI":"10.1016\/S2589-7500(21)00106-0","volume":"3","author":"JCY Seah","year":"2021","unstructured":"Seah, J. C. Y. et al. Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: A retrospective, multireader multicase study. Lancet Digit. Heal. 3, e496\u2013e506 (2021).","journal-title":"Lancet Digit. Heal."},{"key":"658_CR21","doi-asserted-by":"publisher","first-page":"e1002686","DOI":"10.1371\/journal.pmed.1002686","volume":"15","author":"P Rajpurkar","year":"2018","unstructured":"Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 15, e1002686 (2018).","journal-title":"PLoS Med."},{"key":"658_CR22","doi-asserted-by":"crossref","unstructured":"Nam, J. G. et al. Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur. Respir. J. 57, (2021).","DOI":"10.1183\/13993003.03061-2020"},{"key":"658_CR23","doi-asserted-by":"publisher","first-page":"e543","DOI":"10.1016\/S2589-7500(21)00116-3","volume":"3","author":"ZZ Qin","year":"2021","unstructured":"Qin, Z. Z. et al. Tuberculosis detection from chest x-rays for triaging in a high tuberculosis-burden setting: An evaluation of five artificial intelligence algorithms. Lancet Digit. Heal. 3, e543\u2013e554 (2021).","journal-title":"Lancet Digit. Heal."},{"key":"658_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s00330-021-08074-7","volume":"31","author":"H Yoo","year":"2021","unstructured":"Yoo, H. et al. AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset. Eur. Radiol. 31, 1\u201311 (2021).","journal-title":"Eur. Radiol."},{"key":"658_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12885-021-08847-9","volume":"21","author":"D Ueda","year":"2021","unstructured":"Ueda, D. et al. Artificial intelligence-supported lung cancer detection by multi-institutional readers with multi-vendor chest radiographs: a retrospective clinical validation study. BMC Cancer 21, 1\u20138 (2021).","journal-title":"BMC Cancer"},{"key":"658_CR26","doi-asserted-by":"publisher","first-page":"1390","DOI":"10.1093\/cid\/ciab639","volume":"74","author":"G Tavaziva","year":"2021","unstructured":"Tavaziva, G. et al. Chest X-ray analysis with deep learning-based software as a triage test for pulmonary tuberculosis: An individual patient data meta-analysis of diagnostic accuracy. Clin. Infect. Dis. 74, 1390\u20131400 (2021).","journal-title":"Clin. Infect. Dis."},{"key":"658_CR27","doi-asserted-by":"publisher","first-page":"e573","DOI":"10.1016\/S2589-7500(20)30221-1","volume":"2","author":"FA Khan","year":"2020","unstructured":"Khan, F. A. et al. Chest x-ray analysis with deep learning-based software as a triage test for pulmonary tuberculosis: A prospective study of diagnostic accuracy for culture-confirmed disease. Lancet Digit. Heal. 2, e573\u2013e581 (2020).","journal-title":"Lancet Digit. Heal."},{"key":"658_CR28","doi-asserted-by":"publisher","first-page":"e2141096","DOI":"10.1001\/jamanetworkopen.2021.41096","volume":"4","author":"F Homayounieh","year":"2021","unstructured":"Homayounieh, F. et al. An artificial intelligence\u2013based chest X-ray model on human nodule detection accuracy from a multicenter study. JAMA Netw. Open 4, e2141096 (2021).","journal-title":"JAMA Netw. Open"},{"key":"658_CR29","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","volume":"3","author":"M Roberts","year":"2021","unstructured":"Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3, 199\u2013217 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"658_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-021-00438-z","volume":"4","author":"R Aggarwal","year":"2021","unstructured":"Aggarwal, R. et al. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. npj Digital Med. 4, 1\u201323 (2021).","journal-title":"npj Digital Med."},{"key":"658_CR31","doi-asserted-by":"publisher","unstructured":"Lu, J. H. et al. Low adherence to existing model reporting guidelines by commonly used clinical prediction models. medRxiv https:\/\/doi.org\/10.1101\/2021.07.21.21260282 (2021).","DOI":"10.1101\/2021.07.21.21260282"},{"key":"658_CR32","first-page":"50","volume":"38","author":"B Goodman","year":"2017","unstructured":"Goodman, B. & Flaxman, S. European union regulations on algorithmic decision making and a \u2018right to explanation\u2019. AI Mag. 38, 50\u201357 (2017).","journal-title":"AI Mag."},{"key":"658_CR33","unstructured":"FDA. Clinical Performance Assessment: Detection Devices Applied to Radiology Images and Radiology Device Data in - Premarket Notification (510(k)) Submissions Guidance for Industry and FDA Staff. (2020)."},{"key":"658_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41747-020-00159-0","volume":"4","author":"G Baselli","year":"2020","unstructured":"Baselli, G., Codari, M. & Sardanelli, F. Opening the black box of machine learning in radiology: can the proximity of annotated cases be a way? Eur. Radiol. Exp. 4, 1\u20137 (2020).","journal-title":"Eur. Radiol. Exp."},{"key":"658_CR35","unstructured":"Chen, R. J. et al. Algorithm Fairness in AI for Medicine and Healthcare. arXiv preprint arXiv:2110.00603 (2021)."},{"key":"658_CR36","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1006\/ijhc.1999.0252","volume":"51","author":"LJ Skitka","year":"1999","unstructured":"Skitka, L. J., Mosier, K. L. & Burdick, M. Does automation bias decision-making? Int. J. Hum. Comput. Stud. 51, 991\u20131006 (1999).","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"658_CR37","doi-asserted-by":"publisher","first-page":"450","DOI":"10.1148\/radiol.2021202818","volume":"299","author":"J Sung","year":"2021","unstructured":"Sung, J. et al. Added value of deep learning-based detection system for multiple major findings on chest radiographs: A randomized crossover study. Radiology 299, 450\u2013459 (2021).","journal-title":"Radiology"},{"key":"658_CR38","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1007\/s00330-019-06532-x","volume":"30","author":"S Park","year":"2020","unstructured":"Park, S. et al. Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings. Eur. Radiol. 30, 1359\u20131368 (2020).","journal-title":"Eur. Radiol."},{"key":"658_CR39","doi-asserted-by":"publisher","unstructured":"Hong, W. et al. Deep Learning for Detecting Pneumothorax on Chest Radiographs after Needle Biopsy: Clinical Implementation. Radiology https:\/\/doi.org\/10.1148\/radiol.211706 (2022).","DOI":"10.1148\/radiol.211706"},{"key":"658_CR40","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1111\/1754-9485.13105","volume":"65","author":"YH Koo","year":"2021","unstructured":"Koo, Y. H. et al. Extravalidation and reproducibility results of a commercial deep learning-based automatic detection algorithm for pulmonary nodules on chest radiographs at tertiary hospital. J. Med. Imaging Radiat. Oncol. 65, 15\u201322 (2021).","journal-title":"J. Med. Imaging Radiat. Oncol."},{"key":"658_CR41","doi-asserted-by":"publisher","first-page":"1069","DOI":"10.1007\/s00330-020-07219-4","volume":"31","author":"JH Lee","year":"2021","unstructured":"Lee, J. H. et al. Deep learning\u2013based automated detection algorithm for active pulmonary tuberculosis on chest radiographs: Diagnostic performance in systematic screening of asymptomatic individuals. Eur. Radiol. 31, 1069\u20131080 (2021).","journal-title":"Eur. Radiol."},{"key":"658_CR42","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1148\/radiol.2019191225","volume":"293","author":"EJ Hwang","year":"2019","unstructured":"Hwang, E. J. et al. Deep learning for chest radiograph diagnosis in the emergency department. Radiology 293, 573\u2013580 (2019).","journal-title":"Radiology"},{"key":"658_CR43","doi-asserted-by":"publisher","first-page":"e25663","DOI":"10.1097\/MD.0000000000025663","volume":"100","author":"SY Choi","year":"2021","unstructured":"Choi, S. Y. et al. Evaluation of a deep learning-based computer-aided detection algorithm on chest radiographs: Case-control study. Med. (Baltim.) 100, e25663 (2021).","journal-title":"Med. (Baltim.)"},{"key":"658_CR44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-021-93967-2","volume":"11","author":"Z Nabulsi","year":"2021","unstructured":"Nabulsi, Z. et al. Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Sci. Rep. 11, 1\u201315 (2021).","journal-title":"Sci. Rep."},{"key":"658_CR45","doi-asserted-by":"publisher","first-page":"e0246472","DOI":"10.1371\/journal.pone.0246472","volume":"16","author":"EY Kim","year":"2021","unstructured":"Kim, E. Y. et al. Performance of a deep-learning algorithm for referable thoracic abnormalities on chest radiographs: A multicenter study of a health screening cohort. PLoS One 16, e0246472 (2021).","journal-title":"PLoS One"},{"key":"658_CR46","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1186\/1471-2288-14-40","volume":"14","author":"GS Collins","year":"2014","unstructured":"Collins, G. S. et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med. Res. Methodol. 14, 40 (2014).","journal-title":"BMC Med. Res. Methodol."},{"key":"658_CR47","doi-asserted-by":"publisher","first-page":"e200029","DOI":"10.1148\/ryai.2020200029","volume":"2","author":"J Mongan","year":"2020","unstructured":"Mongan, J., Moy, L. & Kahn, C. E. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A guide for authors and reviewers. Radiol. Artif. Intell. 2, e200029 (2020).","journal-title":"Radiol. Artif. Intell."},{"key":"658_CR48","doi-asserted-by":"publisher","first-page":"W1","DOI":"10.7326\/M14-0698","volume":"162","author":"KGM Moons","year":"2015","unstructured":"Moons, K. G. M. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration. Ann. Intern. Med. 162, W1\u2013W73 (2015).","journal-title":"Ann. Intern. Med."},{"key":"658_CR49","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1161\/CIRCULATIONAHA.114.014508","volume":"131","author":"GS Collins","year":"2015","unstructured":"Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) the TRIPOD statement. Circulation 131, 211\u2013219 (2015).","journal-title":"Circulation"},{"key":"658_CR50","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1177\/0272989X14547233","volume":"35","author":"B Van Calster","year":"2015","unstructured":"Van Calster, B. & Vickers, A. J. Calibration of risk prediction models: Impact on decision-analytic performance. Med. Decis. Mak. 35, 162\u2013169 (2015).","journal-title":"Med. Decis. Mak."},{"key":"658_CR51","doi-asserted-by":"publisher","first-page":"3660","DOI":"10.1007\/s00330-020-06771-3","volume":"30","author":"EJ Hwang","year":"2020","unstructured":"Hwang, E. J. et al. Deep learning algorithm for surveillance of pneumothorax after lung biopsy: a multicenter diagnostic cohort study. Eur. Radiol. 30, 3660\u20133671 (2020).","journal-title":"Eur. Radiol."},{"key":"658_CR52","doi-asserted-by":"crossref","unstructured":"Saporta, A. Benchmarking saliency methods for chest X-ray interpretation. medRxiv 2021.02.28.21252634 (2021).","DOI":"10.1101\/2021.02.28.21252634"},{"key":"658_CR53","doi-asserted-by":"crossref","unstructured":"Seyyed-Kalantari, L., Liu, G., Mcdermott, M., Chen, I. Y. & Ghassemi, M. CheXclusion: Fairness gaps in deep chest X-ray classifiers. BIOCOMPUTING 2021: proceedings of the Pacific symposium, pp. 232\u2013243 (2020).","DOI":"10.1142\/9789811232701_0022"},{"key":"658_CR54","doi-asserted-by":"publisher","first-page":"e1002683","DOI":"10.1371\/journal.pmed.1002683","volume":"15","author":"JR Zech","year":"2018","unstructured":"Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Med 15, e1002683 (2018).","journal-title":"PLOS Med"},{"key":"658_CR55","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster, B. et al. Calibration: The Achilles heel of predictive analytics. BMC Med. 17, 230 (2019).","journal-title":"BMC Med."},{"key":"658_CR56","doi-asserted-by":"publisher","first-page":"6902","DOI":"10.1007\/s00330-020-07062-7","volume":"30","author":"EJ Hwang","year":"2020","unstructured":"Hwang, E. J., Kim, H., Lee, J. H., Goo, J. M. & Park, C. M. Automated identification of chest radiographs with referable abnormality with deep learning: need for recalibration. Eur. Radiol. 30, 6902\u20136912 (2020).","journal-title":"Eur. Radiol."},{"key":"658_CR57","doi-asserted-by":"publisher","first-page":"1981","DOI":"10.3390\/jcm9061981","volume":"9","author":"JH Kim","year":"2020","unstructured":"Kim, J. H. et al. Clinical Validation of a Deep Learning Algorithm for Detection of Pneumonia on Chest Radiographs in Emergency Department Patients with Acute Febrile Respiratory Illness. J. Clin. Med. 9, 1981 (2020).","journal-title":"J. Clin. Med."},{"key":"658_CR58","unstructured":"Vafaei, A., Hatamabadi, H. R., Heidary, K., Alimohammadi, H. & Tarbiyat, M. Diagnostic accuracy of ultrasonography and radiography in initial evaluation of chest trauma patients. Emergency vol. 4 www.jemerg.com (2016)."},{"key":"658_CR59","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-0232-8","volume":"3","author":"A Kiani","year":"2020","unstructured":"Kiani, A. et al. Impact of a deep learning assistant on the histopathologic classification of liver cancer. npj Digital Med. 3, 1\u20138 (2020).","journal-title":"npj Digital Med."},{"key":"658_CR60","first-page":"PA4820","volume":"54","author":"AK Prakash","year":"2019","unstructured":"Prakash, A. K. et al. To evaluate the inter and intraobserver agreement in the initial diagnosis by digital chest radiograph sent via whatsapp messenger. Eur. Respir. J. 54, PA4820 (2019).","journal-title":"Eur. Respir. J."},{"key":"658_CR61","doi-asserted-by":"publisher","first-page":"1710","DOI":"10.5858\/arpa.2013-0093-CP","volume":"137","author":"L Pantanowitz","year":"2013","unstructured":"Pantanowitz, L. et al. Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the college of american pathologists pathology and laboratory quality center. Arch. Pathol. Lab. Med. 137, 1710 (2013).","journal-title":"Arch. Pathol. Lab. Med."},{"key":"658_CR62","first-page":"1364","volume":"370","author":"X Liu","year":"2020","unstructured":"Liu, X., Rivera, S. C., Moher, D., Calvert, M. J. & Denniston, A. K. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI Extension. BMJ 370, 1364\u20131374 (2020).","journal-title":"BMJ"},{"key":"658_CR63","doi-asserted-by":"publisher","first-page":"e1","DOI":"10.1016\/j.jclinepi.2010.03.004","volume":"63","author":"D Moher","year":"2010","unstructured":"Moher, D. et al. CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. J. Clin. Epidemiol. 63, e1\u2013e37 (2010).","journal-title":"J. Clin. Epidemiol."},{"key":"658_CR64","doi-asserted-by":"publisher","first-page":"698","DOI":"10.1136\/bmj.c332","volume":"340","author":"KF Schulz","year":"2010","unstructured":"Schulz, K. F., Altman, D. G. & Moher, D. CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. BMJ 340, 698\u2013702 (2010).","journal-title":"BMJ"},{"key":"658_CR65","doi-asserted-by":"publisher","first-page":"h5527","DOI":"10.1136\/bmj.h5527","volume":"351","author":"PM Bossuyt","year":"2015","unstructured":"Bossuyt, P. M. et al. STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies. BMJ 351, h5527 (2015).","journal-title":"BMJ"},{"key":"658_CR66","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1136\/bmjopen-2016-012799","volume":"6","author":"JF Cohen","year":"2016","unstructured":"Cohen, J. F. et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: Explanation and elaboration. BMJ Open 6, 1\u201317 (2016).","journal-title":"BMJ Open"},{"key":"658_CR67","unstructured":"R Core Team. R: A language and environment for statistical computing. (2020)."},{"key":"658_CR68","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","volume":"26","author":"CJ Clopper","year":"1934","unstructured":"Clopper, C. J. & Pearson, E. S. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika 26, 404\u2013413 (1934).","journal-title":"Biometrika"},{"key":"658_CR69","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1016\/j.jclinepi.2004.12.009","volume":"58","author":"A Flahault","year":"2005","unstructured":"Flahault, A., Cadilhac, M. & Thomas, G. Sample size calculation should be performed for design accuracy in diagnostic test studies. J. Clin. Epidemiol. 58, 859\u2013862 (2005).","journal-title":"J. Clin. Epidemiol."},{"key":"658_CR70","unstructured":"Deepnoid. DEEP:LABEL. (2020)."},{"key":"658_CR71","doi-asserted-by":"publisher","first-page":"297","DOI":"10.2307\/1932409","volume":"26","author":"LR Dice","year":"1945","unstructured":"Dice, L. R. Measures of the Amount of Ecologic Association Between Species. Ecology 26, 297\u2013302 (1945).","journal-title":"Ecology"},{"key":"658_CR72","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/S1076-6332(03)00671-8","volume":"11","author":"KH Zou","year":"2004","unstructured":"Zou, K. H. et al. Statistical Validation of Image Segmentation Quality Based on a Spatial Overlap Index. Acad. Radiol. 11, 178\u2013189 (2004).","journal-title":"Acad. Radiol."},{"key":"658_CR73","doi-asserted-by":"publisher","first-page":"1427","DOI":"10.1148\/rg.2021200210","volume":"41","author":"PM Cheng","year":"2021","unstructured":"Cheng, P. M. et al. Deep learning: An update for radiologists. Radiographics 41, 1427\u20131445 (2021).","journal-title":"Radiographics"},{"key":"658_CR74","doi-asserted-by":"publisher","first-page":"4529","DOI":"10.1002\/sim.7179","volume":"36","author":"Y Vergouwe","year":"2017","unstructured":"Vergouwe, Y. et al. A closed testing procedure to select an appropriate method for updating prediction models. Stat. Med. 36, 4529\u20134539 (2017).","journal-title":"Stat. Med."},{"key":"658_CR75","doi-asserted-by":"publisher","first-page":"ii18","DOI":"10.1136\/thx.2010.136986","volume":"65","author":"A MacDuff","year":"2010","unstructured":"MacDuff, A., Arnold, A. & Harvey, J. Management of spontaneous pneumothorax: British Thoracic Society pleural disease guideline 2010. Thorax 65, ii18\u2013ii31 (2010).","journal-title":"Thorax"},{"key":"658_CR76","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1002\/sim.2331","volume":"25","author":"P Royston","year":"2006","unstructured":"Royston, P., Altman, D. G. & Sauerbrei, W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat. Med. 25, 127\u2013141 (2006).","journal-title":"Stat. Med."},{"key":"658_CR77","doi-asserted-by":"publisher","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","volume":"66","author":"A Bustos","year":"2020","unstructured":"Bustos, A., Pertusa, A., Salinas, J. M. & de la Iglesia-Vay\u00e1, M. PadChest: A large chest x-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020).","journal-title":"Med. Image Anal."},{"key":"658_CR78","doi-asserted-by":"crossref","unstructured":"Irvin, J. et al. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In 33rd AAAI Conference on Artificial Intelligence 590\u2013597 (AAAI Press, 2019).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"658_CR79","unstructured":"Asan Image Metrics & Medicallogic. AiCRO System. (2017)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00658-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00658-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00658-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T08:11:56Z","timestamp":1669363916000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00658-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,30]]},"references-count":79,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["658"],"URL":"https:\/\/doi.org\/10.1038\/s41746-022-00658-x","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,30]]},"assertion":[{"value":"24 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This work was supported by a grant from Deepnoid, Inc. S.Y.L., S.H., M.G.J., H.L., H.P.K., H.C., and J.M.C. were employed by Deepnoid, Inc. All the other authors declared no potential conflicts of interest with respect to the research.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"107"}}